Upscaling Infinity Engine Maps with AI
Rhiyo
Member Posts: 6
Upscaling NN algorithms have recently became popular and there are some very successful examples with Morrowind and DOOM.
Me and some others the gameupscale subreddit have been experimenting with some different games and I decided to have a go at upscaling some Infinity Engine maps.
Original (Upscaled x4):
Upscaled by AI:
More with BG2. and more less successful attempts with IWD.
Some of them didn't work out as well as I hoped but it's all about experimentation. I also used Waifu2x when I think I would get better results using ESRGAN, but I don't have the GPU to use ESRGAN with the size of these images. Waifu2x was trained on anime imagery which obviously doesn't translate well to the type of maps in Infinity Engine games.
Infinity Engine games mostly still look good today but I think they could still gain a slight benefit from upscaling, especially if you play with enhanced editions and use the zoom feature. We've been experimenting with other games at r/gameupscale as I think these upscaling algorithms have the potential to improve a lot of older games
I couldn't get them working properly in game though, but I've never worked with modding Infinity Engine games before. I think I'm incorrectly converting them back to TIF as the seams become very apparent.
Me and some others the gameupscale subreddit have been experimenting with some different games and I decided to have a go at upscaling some Infinity Engine maps.
Original (Upscaled x4):
Upscaled by AI:
More with BG2. and more less successful attempts with IWD.
Some of them didn't work out as well as I hoped but it's all about experimentation. I also used Waifu2x when I think I would get better results using ESRGAN, but I don't have the GPU to use ESRGAN with the size of these images. Waifu2x was trained on anime imagery which obviously doesn't translate well to the type of maps in Infinity Engine games.
Infinity Engine games mostly still look good today but I think they could still gain a slight benefit from upscaling, especially if you play with enhanced editions and use the zoom feature. We've been experimenting with other games at r/gameupscale as I think these upscaling algorithms have the potential to improve a lot of older games
I couldn't get them working properly in game though, but I've never worked with modding Infinity Engine games before. I think I'm incorrectly converting them back to TIF as the seams become very apparent.
Post edited by Rhiyo on
18
Comments
I downloaded Python 3.7 and Windows visual studio. After that I installed the Cuda toolkit 10. The problem I am having is with pytorch. When I type the commands into my command prompt it says that 'pip3' is not recognized as an internal or external command. So I'm stuck. I also downloaded the ESRGAN project to my machine.
I think this would totally fix the pixilation on the sprite characters using the nearest neighbor scaling. It would also drastically increase perceived detail on the backdrops of BG1EE. It would also fix pixilation issues on some of the pits of BG2EE. Beamdog should look into this as it would provide a large increase to overall clarity if it were done at 8X. Looks like Nvidia has a beta program for game developers to use up to 8X scaling on their own AI learning algorithm.
@JuliusBorisov you may want to drop Trent this link and see what he thinks. https://developer.nvidia.com/gwmt
This is my first attempt and I am by no means a digital artist so I am going to have to mess with it for a while but I think I could help enhance the backdrops into a more HD looking overall image. I have not attempted to de pixelate the sprite character models but considering how many monsters there are and all of the various frames that make up their animations....It would probably require that a team of users work together. Either that or a few of us spend a whole year doing it.
Nvidia already has a nice DSR solution in their control panel but Beamdog would have to add a resolution selection option in the game for us to take advantage of it. If the engine can work with higher resolution images somehow, you would see a lot more improvement than what is being shown here. Seeing them on my desktop upconverted is a much more substantial improvement.
Edit:
I removed the images as the AI upscale isn't working correctly due to the size of the image. I'm going to attempt to work around the problem to see if I can get something a little more beneficial.
This is nothing the community can do on their own, but @Beamdog, I'd re-buy all the games at least on windows, maybe android, at full price for a decently upscaled 4k edition. just saying.
Here is the original backdrop:
Here is that same backdrop after being upscaled 600% using AI and then downscaled back down to the same resolution, check the floor and walls for added textures and clarity:
You should be able to see an uptick in sharpness. The mugs on the tables are more clear, some detail was added to the floor and walls but I noticed a loss of detail on the bed rolls and a slight increase in overall brightness. So I took the AI enhanced backdrop and applied a mask to it. I then increased fine details and sharpness just to the tables, bedrolls, and the pile of logs at the back of the room. I then tweaked brightness and shadows to mimic the original look the best I could.
So here is a screen shot of the original backdrop in game at 4K with the camera zoomed in 8 clicks:
Here is the same shot with the AI upscale backdrop with added tweaks:
You should be able to see added clarity and more detail. The image looks very nice at 6x which is too large to post here but it can't be used in game anyway. Click on them and open them at full screen to get a better idea. The problem is that many of the backdrops in Baldur's Gate are huge and I am not seeing this kind of detail increase on the main maps. I could make much larger changes to these backdrops but my intention is to leave the contrast, lighting, and overall appearance of the image as close to the original intent as possible. Increasing the details or sharpness too much will cause very visible flaws in the image that look out of place. The UI and sprite character models should look exactly the same because I have made no attempts to upscale either of them.
I do have a few ideas for improving the main maps but it will have to wait until I have time.
Again, if the engine could be modified to use larger upscaled backdrops we would see rather large gains but I don't know how to do it. Maybe it's possible but as of now the only thing I can do is upscale them and then downscale them= worse image than the one I started with minus a few minor detail benefits. It takes a lot of work just to get the image back to it's original quality levels with those added details and sometimes there is no benefit from the AI upscale at all once it's been downscaled again.
Like I said before, most of the gains I am seeing is the result of me going in and hand altering little details.(time consuming) it can be done with better results than before using todays software but it really just comes down to spending the time to adjust the image by hand which is something that has been possible for a long time now. Anyway, here are a few more areas I have been working on. Due to the camera zoom, you will need to click on these to actually see the difference.
Original:
Upscaled:
Original:
Upscaled:
In case it couldn’t be made official, what would the challenges be for making it into a mod? Besides the tedium of having to upscale a ridiculous amount of backdrops.
Also, can the same be done with sprites and UI assets, potentially?
I'm not going to lie though. I don't understand the game engine as I am just a regular joe. I may be able to figure it out with trial and error but it will take a long time. It took me days to figure out the sprites. For example, the animation of your character casting a spell is made up of 270 individual pictures. Gigapixel and ERSGAN are batch file up converters so you can use near infinity to export the .BAM into graphics PNG. Then drag all 270 images into the upscaler and it will increase all 270 frames by 600% in about 2 minutes. After that you need to use near infinity to convert them back to .BAM and Beamdog has it's own version of BAMS that only work on the EE. The massive number of .BAM files and Backdrops would take a very long time even for a team of people to do. It's a massive amount of work.
I would be willing to power through it and let my machine run all day processing batches of files but I just don't have the modding knowledge to do anything efficiently. If there was documentation somewhere where I could read how to upgrade a map 6X and re insert .BAM files without error I would be up for it. Either way, I think this modification would require a group effort.
Here is another problem, 6X scaling is not possible unless you want the game to take up 10TB of hardrive space. Just candle keep at 6X was 1.3GB for the backdrop. Again this is out of my experience, maybe they could be crushed down into smaller file sizes or we would just have to deal with 2X upscaling. I dont' see it happening unless Beamdog does it. They are the only ones that have the knowledge to quickly setup the appropriate settings for the engine. Little, things like animated fire places are separate from the backgrounds and when I re-insert backgrounds they fall out of place with the animated objects like doors, fire pits, ect. It's very touchy and annoying. I would love to see this happen as well but I have to be honest, after messing with it for a while, I don't expect Beamdog to do this even if it is possible (I believe it is). Besides they have already sold the EE titles to the masses, they have little incentive to put forth such a massive effort.
I just started using near infinity for the first time. There are people who understand NI and have been around modding the game since it released. They would be in a better position than I am to say if this is a realistic project. Upscaling an image is a simple task but the Infinity Engine is very annoying to work with. I've worked on stalker graphics mods in the past and that was a breeze to work with in comparison.
You take for example the inventory background image 'INVENTOR' mos file and upscale it to 4k. Then, you search the UI.MENU file for the "INVENTOR" reference and type scaleToClip underneath it-
label { area 0 0 864 710 mosaic INVENTOR scaleToClip }scaleToClip forces the image here to resize into the 864x710 size. The real benefit there is for the scaled interface mode, where the engine "changes" your in-game resolution to 768p so if you have for example full HD monitor, it upscales all images from 768p to 1080p and causes loss of quality. With scaleToClip and 4k image, it resizes that 4k image into 2k, thus quality is preserved.
So what would need to be done for UHD UI is to upscale all UI images among MOS and BAM files and insert scaleToClip into UI.MENU for all UI elements that use images (mosaic and bam).
There are a few obstacles, and the team is deciding on the course of action.
For the UI mod I'm currently working on, all graphic assets are being upscaled/edited/made from scratch with 2160p screens in mind. But since scaleToClip is limited to BAMs, what I'm doing is then downscaling elements again to half their size (so they don't look too big on 1080p or lower) and setting aside the larger files for an eventual 4K-compatible version.
The best case scenario would be if Beamdog made BG(2)/IWD:EE behave the same way PST:EE does, in which the UI is made for 2160p and automatically downscaled according to current screen resolution.
Until then, the best we can do is to have a 4K UI that will only look good in that resolution. And even then, that doesn't solve the problem of the game world itself being way too small (and way too blurred when zooming in).
I really hope they will consider doing it before dropping development of these games because it would pretty much make them future-proof. I don't know how long until 4K screens become the norm, but I'm pretty sure there'll still be people playing these games by then.
the results are not perfect, but quite great anyway. can't imagine anything better really.
It actually works in the game, but using scaleToClip screwed up the image. Check it out:
This is because the resulting PVRZ file from the conversion seems to be capped at 1024 x 1024. So there might be a limit to the size of BAMs after all.
It seems modding isn't going to cut it when making the game enjoyable at 4K resolution. It's going to take engine-level tweaking, which at least PST:EE proves to be possible. The question is whether the devs are willing to backport such functionality to BG (and possibly IWD).
I started in on this because I was simply interested in it. I had fun and learned some new things but it can really be annoying sometimes. Made me respect the guys at Beamdog and the amount of work they have put in to do what they have. SOD must have been a lot of work.