Skip to content

Upscaling Infinity Engine Maps with AI

RhiyoRhiyo Member Posts: 6
Upscaling NN algorithms have recently became popular and there are some very successful examples with Morrowind and DOOM.

Me and some others the gameupscale subreddit have been experimenting with some different games and I decided to have a go at upscaling some Infinity Engine maps.

Original (Upscaled x4):


Upscaled by AI:


More with BG2. and more less successful attempts with IWD.

Some of them didn't work out as well as I hoped but it's all about experimentation. I also used Waifu2x when I think I would get better results using ESRGAN, but I don't have the GPU to use ESRGAN with the size of these images. Waifu2x was trained on anime imagery which obviously doesn't translate well to the type of maps in Infinity Engine games.

Infinity Engine games mostly still look good today but I think they could still gain a slight benefit from upscaling, especially if you play with enhanced editions and use the zoom feature. We've been experimenting with other games at r/gameupscale as I think these upscaling algorithms have the potential to improve a lot of older games :)

I couldn't get them working properly in game though, but I've never worked with modding Infinity Engine games before. I think I'm incorrectly converting them back to TIF as the seams become very apparent.
Post edited by Rhiyo on
ALIENArunsunJuliusBorisovFlashburnAndreaColomboSethDavisthe_sexteingorgonzolaMateoFrozenBubbAedanKamigoroshiAdulhistamiini
«1

Comments

  • ALIENALIEN Member Posts: 1,270
    edited December 2018
    It really is a time of miracles.
    RhiyoJuliusBorisovBubbbob_veng
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I recently checked out the AI upscaled screenshots of Final Fantasy 7 and Doom. I would love to see the backgrounds for Baldur's Gate upgraded. I use a definition enhancer on my TV which makes them look much higher quality than they are but I think this would be even greater of a benefit. I have 3 high end GPU's in my machine (3X Water Cooled 1080TIs), any tips on where to get started? I was reading about these AI upscales a couple of days ago over at hardocp and would love to give it a try. I would be glad to help if you need some extra processing power.

    I downloaded Python 3.7 and Windows visual studio. After that I installed the Cuda toolkit 10. The problem I am having is with pytorch. When I type the commands into my command prompt it says that 'pip3' is not recognized as an internal or external command. So I'm stuck. I also downloaded the ESRGAN project to my machine.

    I think this would totally fix the pixilation on the sprite characters using the nearest neighbor scaling. It would also drastically increase perceived detail on the backdrops of BG1EE. It would also fix pixilation issues on some of the pits of BG2EE. Beamdog should look into this as it would provide a large increase to overall clarity if it were done at 8X. Looks like Nvidia has a beta program for game developers to use up to 8X scaling on their own AI learning algorithm.

    @JuliusBorisov you may want to drop Trent this link and see what he thinks. https://developer.nvidia.com/gwmt
    Post edited by the_sextein on
    JuliusBorisovAndreaColombo
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I've decided to erase this comment. For one, I don't want to discourage anyone from trying to improve this game regardless of the method. Secondly, I have found some improvements that can be had that some may consider worth while. Lastly, I'm just starting to get my feet wet with this stuff and I may be able to find ways to improve the backdrops beyond what I was seeing with my initial tests.
    Post edited by the_sextein on
    JuliusBorisovArdul
  • BelegCuthalionBelegCuthalion Member, Mobile Tester Posts: 453
    It would be an improvement though if the engine would actually support larger background images, and have real 4k/5k/Retina/hidpi/...-support (not just work with it without bugs, but actually take advantage of it by matching zoom and UI scaling and using those extra pixels from the source and on the screen).

    This is nothing the community can do on their own, but @Beamdog, I'd re-buy all the games at least on windows, maybe android, at full price for a decently upscaled 4k edition. just saying.
  • RaduzielRaduziel Member Posts: 4,714
    I think I'm the only one who can't see any difference between the images.
    Noon
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    The purpose of upscaling is to make small images larger or to allow you to zoom in further without seeing the pixels. If the game engine could use larger images it would allow for some impressive gains but without it... I have to down scale the image after I upscale it. Down scaling causes quality loss and it can make the image look worse than it did originally. From what I am experiencing, the AI upscaler will remove noise and blur. Sometimes it can make the image look flat as a result. It's worth experimenting with because sometimes the AI will bring out textures to the surface or add to them, but most of the time I am seeing little or no benefits from it.

    Again, if the engine could be modified to use larger upscaled backdrops we would see rather large gains but I don't know how to do it. Maybe it's possible but as of now the only thing I can do is upscale them and then downscale them= worse image than the one I started with minus a few minor detail benefits. It takes a lot of work just to get the image back to it's original quality levels with those added details and sometimes there is no benefit from the AI upscale at all once it's been downscaled again.

    Like I said before, most of the gains I am seeing is the result of me going in and hand altering little details.(time consuming) it can be done with better results than before using todays software but it really just comes down to spending the time to adjust the image by hand which is something that has been possible for a long time now. Anyway, here are a few more areas I have been working on. Due to the camera zoom, you will need to click on these to actually see the difference.

    Original:


    Upscaled:




    Original:



    Upscaled:




    Post edited by the_sextein on
    JuliusBorisovAndreaColomboKamigoroshi
  • AndreaColomboAndreaColombo Member Posts: 5,524
    edited January 2019
    I really hope this can be a thing for the EEs.

    In case it couldn’t be made official, what would the challenges be for making it into a mod? Besides the tedium of having to upscale a ridiculous amount of backdrops.

    Also, can the same be done with sprites and UI assets, potentially?
    Kamigoroshi
  • PeccaPecca Member Posts: 2,175
    edited January 2019

    I really hope this can be a thing for the EEs.

    In case it couldn’t be made official, what would the challenges be for making it into a mod? Besides the tedium of having to upscale a ridiculous amount of backdrops.

    Also, can the same be done with sprites and UI assets, potentially?

    I think it can be used for UI to display sharp UHD images on scaled interface for example. Since 2.5 there is new functionality for that.
    AndreaColombo
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I've upscaled a few .BAM files to 6X scale and placed them back in the game. It makes your character larger than the candle keep inn. The backdrops can be scaled 6X and placed back in the game as well but the problem is that each map has 3 layers of information and if they don't match then it cuts the upscaled backdrop up and places different parts of the image all over the screen. It may be possible to increase the size of the two other layers to match the new upscaled backdrop. Basically, if you upscale the sprites and the backdrops and re insert them into the game without graphical flaws then you could pull the camera back 600% and it would have the same look but way sharper and more detailed. You could zoom all the way in without pixilation.

    I'm not going to lie though. I don't understand the game engine as I am just a regular joe. I may be able to figure it out with trial and error but it will take a long time. It took me days to figure out the sprites. For example, the animation of your character casting a spell is made up of 270 individual pictures. Gigapixel and ERSGAN are batch file up converters so you can use near infinity to export the .BAM into graphics PNG. Then drag all 270 images into the upscaler and it will increase all 270 frames by 600% in about 2 minutes. After that you need to use near infinity to convert them back to .BAM and Beamdog has it's own version of BAMS that only work on the EE. The massive number of .BAM files and Backdrops would take a very long time even for a team of people to do. It's a massive amount of work.

    I would be willing to power through it and let my machine run all day processing batches of files but I just don't have the modding knowledge to do anything efficiently. If there was documentation somewhere where I could read how to upgrade a map 6X and re insert .BAM files without error I would be up for it. Either way, I think this modification would require a group effort.

    Here is another problem, 6X scaling is not possible unless you want the game to take up 10TB of hardrive space. Just candle keep at 6X was 1.3GB for the backdrop. Again this is out of my experience, maybe they could be crushed down into smaller file sizes or we would just have to deal with 2X upscaling. I dont' see it happening unless Beamdog does it. They are the only ones that have the knowledge to quickly setup the appropriate settings for the engine. Little, things like animated fire places are separate from the backgrounds and when I re-insert backgrounds they fall out of place with the animated objects like doors, fire pits, ect. It's very touchy and annoying. I would love to see this happen as well but I have to be honest, after messing with it for a while, I don't expect Beamdog to do this even if it is possible (I believe it is). Besides they have already sold the EE titles to the masses, they have little incentive to put forth such a massive effort.

    I just started using near infinity for the first time. There are people who understand NI and have been around modding the game since it released. They would be in a better position than I am to say if this is a realistic project. Upscaling an image is a simple task but the Infinity Engine is very annoying to work with. I've worked on stalker graphics mods in the past and that was a breeze to work with in comparison.
    KamigoroshiKilivitzAndreaColombo
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    Pecca said:

    I really hope this can be a thing for the EEs.

    In case it couldn’t be made official, what would the challenges be for making it into a mod? Besides the tedium of having to upscale a ridiculous amount of backdrops.

    Also, can the same be done with sprites and UI assets, potentially?

    I think it can be used for UI to display sharp UHD images on scaled interface for example. Since 2.5 there is new functionality for that.
    I upscaled the UI to 2160 height for natural 4K 2160P visuals. This causes the the image to get fatter naturally. You have to then tell the UI where to place the image because the positioning is set up for smaller textures. Same thing is happening with the backdrops. You need to tell the backdrops where to position but it's a little more complicated then that based on my experiences with it. Upscaling in game won't work because it does it's own version of it on the fly. So the UI will need to be reworked for placement of the larger upscaled textures. It does work though.
    Isandir
  • PeccaPecca Member Posts: 2,175
    edited January 2019
    Though I haven't tested myself I think it works like this:
    You take for example the inventory background image 'INVENTOR' mos file and upscale it to 4k. Then, you search the UI.MENU file for the "INVENTOR" reference and type scaleToClip underneath it-

    label { area 0 0 864 710 mosaic INVENTOR scaleToClip }

    scaleToClip forces the image here to resize into the 864x710 size. The real benefit there is for the scaled interface mode, where the engine "changes" your in-game resolution to 768p so if you have for example full HD monitor, it upscales all images from 768p to 1080p and causes loss of quality. With scaleToClip and 4k image, it resizes that 4k image into 2k, thus quality is preserved.

    So what would need to be done for UHD UI is to upscale all UI images among MOS and BAM files and insert scaleToClip into UI.MENU for all UI elements that use images (mosaic and bam).
    Isandir
  • IsandirIsandir Member Posts: 458
    @the_sextein Thank you taking the time to tackle this. I'd love to see a re-release with HD graphics and would also be willing to buy them again just for that. @JuliusBorisov I'd also be happy to contribute new portraits as well... ;)
    the_sexteinBalrog99
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I've been using Gigapixel which is geared towards photography at 6X. Beamdog could get access to Nvidia's game oriented upscaler at 8X. The results could be exciting. One thing I would recommend checking out is Topaz Studio. It's free for basic and the pro options give you a free trial for 30 days. After that each option is about $50. The reason I mention it is that in my experience, the AI won't always work perfectly. It may clean the image and build sharper textures but sometimes it will diminish textures under certain circumstances. With Topaz I have had luck bringing back most of the lost detail from the AI upscale. Clear AI, Precision Detail, Masking Tools, Precision Contrast and Sharpening are all very useful options that I have used to improve many of the backdrops.
    Isandir
  • IsandirIsandir Member Posts: 458
    After just looking at the website, it's definitely better than other resizing software I've encountered. There have been quite a few cases when I've needed to upscale photos and haven't found any good options. Luckily, I can probably get it through work!
  • KilivitzKilivitz Member Posts: 1,459
    Pecca said:

    Though I haven't tested myself I think it works like this:
    You take for example the inventory background image 'INVENTOR' mos file and upscale it to 4k. Then, you search the UI.MENU file for the "INVENTOR" reference and type scaleToClip underneath it-

    label { area 0 0 864 710 mosaic INVENTOR scaleToClip }

    scaleToClip forces the image here to resize into the 864x710 size. The real benefit there is for the scaled interface mode, where the engine "changes" your in-game resolution to 768p so if you have for example full HD monitor, it upscales all images from 768p to 1080p and causes loss of quality. With scaleToClip and 4k image, it resizes that 4k image into 2k, thus quality is preserved.

    So what would need to be done for UHD UI is to upscale all UI images among MOS and BAM files and insert scaleToClip into UI.MENU for all UI elements that use images (mosaic and bam).

    Unfortunately scaleToClip doesn't seem to work for MOS files, only BAMs.

  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I was trying to upscale directly so that the GUI would look good at 4K without up scaling enabled in the engine. That to me, would be ideal. I unconverted the file height from 1500 pixels to 2160 and that made it widen out. I think it would look good at 4K without the scale user interface option enabled. Unfortunately, the game engine aligns the upscaled texture incorrectly. I would have to go in with the UI editor and toy around which I have not done yet. Have you tried using scale to clip with MOS files that have been converted to PNG?
    Post edited by the_sextein on
  • KilivitzKilivitz Member Posts: 1,459
    edited January 2019
    I've tried it with MOS files I've converted from PNG using NearInfinity.

    For the UI mod I'm currently working on, all graphic assets are being upscaled/edited/made from scratch with 2160p screens in mind. But since scaleToClip is limited to BAMs, what I'm doing is then downscaling elements again to half their size (so they don't look too big on 1080p or lower) and setting aside the larger files for an eventual 4K-compatible version.

    The best case scenario would be if Beamdog made BG(2)/IWD:EE behave the same way PST:EE does, in which the UI is made for 2160p and automatically downscaled according to current screen resolution.

    Until then, the best we can do is to have a 4K UI that will only look good in that resolution. And even then, that doesn't solve the problem of the game world itself being way too small (and way too blurred when zooming in).

    I really hope they will consider doing it before dropping development of these games because it would pretty much make them future-proof. I don't know how long until 4K screens become the norm, but I'm pretty sure there'll still be people playing these games by then.
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    Have you tried converting them into BAMS using the BAM tool in NI? I think you can turn them into PNG and then to a .BAM file. I don't know if it would work as a BAM but it might be worth a try just to see what it would do. You can create a single image .BAM and I think it would display normally without flickering but I've never tried it. I think that's how the character models are displayed in the UI.
  • bob_vengbob_veng Member Posts: 2,308
    i've always thought that beamdog should rescale everything 2x-4x, the whole graphics, using some advanced method, like this one.

    the results are not perfect, but quite great anyway. can't imagine anything better really.
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    Kilivitz said:

    The best case scenario would be if Beamdog made BG(2)/IWD:EE behave the same way PST:EE does, in which the UI is made for 2160p and automatically downscaled according to current screen resolution. blockquote>

    I was thinking the same thing when I was messing with it. I haven't spent much time with the UI but it would make more sense if it were a native 4K UI so it wouldn't need to be stretched at any resolution.

  • KilivitzKilivitz Member Posts: 1,459
    edited January 2019

    Have you tried converting them into BAMS using the BAM tool in NI? I think you can turn them into PNG and then to a .BAM file. I don't know if it would work as a BAM but it might be worth a try just to see what it would do. You can create a single image .BAM and I think it would display normally without flickering but I've never tried it. I think that's how the character models are displayed in the UI.

    I've followed your suggestion and tried turning a background image into a one-cycle, one-frame BAM file. Then I replaced the background mosaic with a button, but it didn't work (game won't display the button, even though the element is there). Then, I tried making it a v2, PVRZ-based bam but the conversion failed. The image is 1362x1280 - there might be a hard-coded limit on the dimensions of BAMs.
  • the_sexteinthe_sextein Member Posts: 711
    Ha ha, it was worth a try.
  • KilivitzKilivitz Member Posts: 1,459
    edited January 2019
    The good news is that I made a mistake when converting the BAM to a v2 (PVRZ-based) file.

    It actually works in the game, but using scaleToClip screwed up the image. Check it out:



    This is because the resulting PVRZ file from the conversion seems to be capped at 1024 x 1024. So there might be a limit to the size of BAMs after all.
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    Maybe you could make a larger image in photo shop or paint and just make it transparent and drop your artwork in the center of it. You then adjust the size of the transparent part of the image until it displays on point in the game. Very trail and error, not to mention annoying but it may work.
  • KilivitzKilivitz Member Posts: 1,459
    edited January 2019
    That could work, but capping window dimensions to 1024x1024 on a screen that's 3840 pixels wide seems like a criminal underuse of screen real estate and it kind of defeats the purpose of a UI that looks comfortable in a 4K monitor, don't you think?

    It seems modding isn't going to cut it when making the game enjoyable at 4K resolution. It's going to take engine-level tweaking, which at least PST:EE proves to be possible. The question is whether the devs are willing to backport such functionality to BG (and possibly IWD).
  • PeccaPecca Member Posts: 2,175
    Then perhaps the 4k image could be split into 1024x1024 bams, that are set next to each other.
    bob_vengthe_sexteinKilivitz
  • the_sexteinthe_sextein Member Posts: 711
    edited January 2019
    I think this is why Beamdog is trying to move away from the Infinity Engine. In a more modern engine you just make what you want and drop it in. This kind of thing is a major pain.

    I started in on this because I was simply interested in it. I had fun and learned some new things but it can really be annoying sometimes. Made me respect the guys at Beamdog and the amount of work they have put in to do what they have. SOD must have been a lot of work.
    JuliusBorisov
Sign In or Register to comment.