Problems with water-overlays in pvrz-files

Acifer

Currently I am converting my original TIS files to PvRZ files, but the water overlays are not converted correctly.
I created the overlays for the „classic“ game, and the water looks good and fills in every gap, even between the planks of the bridge:


After the PVRZ conversion, however, only the "full" tiles appear to be flooded with water.


I don‘t know if I did something wrong during the process of conversion or it is just another pvrz-EE-thing like the horizontal lines that sometimes show up between the tiles.
There was another thread that already mentioned that problem:

kiski wrote: »

But I found that the bridge area in not working as intended in Enhanced edition due to new TIS and PVRZ format - during the conversion in NI water overlay get screwed. Probably I know why, but have no time to correct it, because it would involve to create new "closed" tiles that represent cutted borders around the obstacles like rocks and bridge itself.
I wonder how beamdog converted original areas...

But - I have no idea how to fix it.
Overall, it would be a lot of work to start over from scratch by recreating all these overlays for the EE-Wed file.
Any help or advice would be appreciated!

jastey

*bump* Just in case someone knows an answer, I'd have the same question.

Acifer

Sadly, all efforts to fix the overlay problem did not work as intended.
Even creating a new overlay did not end up well, because DLTCEP converts the pvrz files back to tis again - a neverending vicious circle.

kiski wrote: »

Btw, I finally solved how overlays in Enhanced Edition works. Thus Alandor Gates is corrected now and working in EE like a charm together with water overlay. Both day and night version.

@kiski: How did you manage to solve that water problem?

argent77

I can't really provide a solution, but I can at least outline the problem.

Masked pixels in tileset overlays are defined differently in BG2EE compared to classic BG2. You can see the difference in this example tile (taken from AR0046):


As you can see, secondary tile in BG2 is identical with primary tile in BG2EE.

Primary tile in BG2 requires more effort to fix though. You would have to make a pixel-by-pixel comparison and clear all pixels where secondary tile pixels are not transparent. However, that would require to recreate the tile palette to ensure palette index 0 is defined as "transparent". Otherwise you would see block artefacts in BG2EE.

Acifer

Wow, thank you argent, that helps a lot!

So, my first step would be to "cut out" the tile pixels. I will try to create an object mask in photoshop using a 64X64 grid.

Is there a way to be sure that the palette entry #0 in bitmaps or png's is equivalent to the palette index in the tileset since the palette is dramatically reduced during the bmp->tis-conversion process?
For example by defining the (good old) RGB 255 green color in photoshop as a #0 transparency color?

Otherwise, is there a way to recreate the tile palette in NI?

Again, thank you so much for your help, this problem is haunting me for months.

argent77

NI can convert bitmaps into tilesets. It will consider both transparent or "pure green" RGB(0,255,0) color entries for transparency. The hard part is actually fixing the overlay masks. You would have to scan the associated WED file to find the corresponding primary/secondary tile pairs, unless you want to do everything by hand.

Acifer

A first update: It's a lot of work. But it works!

This is what I tried so far:

- Load the legacy tis in NI, export it to png
- open the png in a photo editor (I tried Photoshop, but ended up in Corel Photopaint due to the better grid)
- duplicate the object, fill the transparent parts with RGB 0,255,0 for better visibility
- Create a 64x64 pixel grid
- Use "snap object to grid"
- cut out the overlay tiles one by one, align them in the primary tile part of the image
- cut out the primary tiles the same way, align them in the overlay tile part
- fill the "legacy" overlay tiles with RGB 255 green color, place them over the new overlay tiles to reverse the transparent parts of the overlay tiles
- I saved it as a bmp, cut out the pure green color, saved it as png and converted it straight to pvrz using NI again.

Here is my progress so far:



As you can see, there are some strange clippings between some tiles. Even though i used a 64x64 grid and snapping, the overlay tiles sometimes do not align well.
I hope the reason is that some overlay tiles do not fit precisely with the primary tiles in the legacy tis-file and I can fix it by hand.

Otherwise it could be another horizontal-line-tile-problem, because it looks pretty good in NI.


I will continue to work on the tis-transformation in the next few days, try to fix the clipping problem and then report back.

Thank you for your great ideas, Argent!

kiski

@Acifer : I did solved it the same way as you. And also have some strange glitches between some tiles, but I´m sure it´s not an overlay problem, cause I saw somewhere that even non-overlated areas have this flicks... Something with the graphic format perhaps, just search the forum for this.

Btw, I think that other way than recreating corresponding tiles by hand won´t be possible. Really can´t imagine how one should automate this proces...

Acifer

@kiski: Thank you for your answers, I'm really glad to have you here!
Some glitches between the tiles seem to come and go when zoomed in and out. So you're right, this could be a pvrz-related thing and not an overlay-problem at all.

However, I encountered some similar artifacts around some prvz-doors.

So I checked some of my areas and realized that the most "abnormal" glitches could be a "design-mistake" during my area-creation process.
I noticed that some pixels of the overlay file do not match with the original tile. In some cases, I re-rendered the area-image using another antialiasing method (Mitchell-Netravali instead of Catmull-Rom) because the image looked too "crispy". Sadly, I did this after the overlay has been finished, so the overlay tiles did not change at all when re-assigning the new tileset.

However, I managed to minimize the ugly lines by assigning a transparent (=pure green) layer below the overlay sector of the tileset. When cut out before the png-conversion, it reduces the lines significantly.
So at the moment I am quite content with the outcome.



The next task will be the pixel-by-pixel adjustment of the water between the bridge.
Right now, I do not have a good idea how to adjust exactly the same pixels in the primary and the overlay tile.

After the complete transformation of this test-area, I will check some more issues and try to create a step-by-step guide for modders with similar problems.

@argent77: thank you again for your great help!

kiski

Acifer wrote: »

Right now, I do not have a good idea how to adjust exactly the same pixels in the primary and the overlay tile.

I believe those corresponding tile should be same but opposite. So on primary background layer overlayed "water" tiles should have corresonding pixels transparent and only a solid border (wall, stone, etc) have something in it, thus non-transparent. On contrary, overlay layer should have borders transparent and water covered background should be non-transparent.

So "overlapped" pixels on both layers must be just the opppsite - if pixel is transparent on primary, on second it must be solid and vice versa.

This setup is only for partially overlayed tiles. Those covered completely are not affected at all and use same satup as for vanilla BG2.

But still have no idea why this was changed in EE games...

argent77

I've taken a closer look at the tileset overlay differences. The best method to convert maps (coding-wise) would be to use the original primary tile and split it into two new tiles (primary and secondary), based on the pixel content of the original secondary tile. Otherwise there may be discrepancies if non-masked content of primary and secondary tiles are not identical. I'll see if I can create a script which performs this conversion.

kiski

@argent77 : But don´t forget that overlay layer must be created first, to determine overlay borders and affected tiles. I´m begging for this for so long, to have such overlay editor in NI. Would it be very hard to implement such feature? In DLTCEP it works, you just make polygon over background map, click Create As Overlay or such, and voila, overlay is done. How hard would be to make such thing for NI? I´m not a programmer, it´s just far beyond my knowledge.

argent77

I'm not talking about something as complex as a full-fledged overlay editor, but rather about a tool to make tilesets created for original BG2 compatible with the Enhanced Editions. In that case the relevant information are already available from the associated WED file.

Acifer

argent77 wrote: »

I'll see if I can create a script which performs this conversion.

That would be a huge help! Thank you.

Would it be possible to "switch" primary with secondary tiles in a wed-file and export that image as png?

So a tileset that looks like this...



...could be exported like that...



When mixing the original and the new exported "swapped" tileset using "floodfill" to the overlay tiles in an image editor you would get the final result really fast:



There are still small gaps visible in the overlay section that have to be adjusted in the image editor, but this would be a good start if it is possible from a coding perspective.

kiski

@argent77 : I know what you mean. I´m just asking if it would be possible to make such feature, to implement something like that. That´s all. Because now we need to use two editors for areas - DLTCEP and NI. Some may say that NI is not needed, but I find NI rather easier to use when switching tilesets, for example... and there´s much more...

@Acifer : Yeah, a nice idea you have here. But don´t forget the doors you must count in, too.

jastey

Just chiming in to say that your area artwork is really beautiful, @Acifer!
So, carry on guys, didn't mean to disturb the constructive discussion.

argent77

@Acifer I have put together a command line tool that performs classic -> EE TIS overlay conversion (see attachment). You'll need both TIS and WED file for the conversion.

I might turn it into a more useful tool that works in both directions when I have the time.

Raduziel

Let me hijack this thread for a second to say how much I admire modders that creates new areas. You guys are fantastic.

Acifer

@argent77: This tool is amazing! It works perfect. Pure, clean water, no more glitches! And it is fast as hell.
Time required to manually convert a single TIS file: 15 hours.
The same TIS conversion in TIS2OVL: 1.6 seconds !!!
I'm so happy - after all these years, I am finally able to proceed with area-editing for the extended editions.

Thank you for your incredible help. This is a huge milestone for the IE-Modding community!

@jastey, @Raduziel:
Thank you so much for your kind words! This means a lot to me. I am very glad that you like my efforts.

LavaDelVortel

@Acifer - is there any thread where you write some more details on what exactly you are working on?

jastey

Just posting here to give @argent77 20 more "likes" for his constant work on tools that make modders' lifes so much easier!

kiski

@argent77 That tool is perfect but I do have glitches... horizontal ones... occasionally, but still. I wonder - PVRZ style tileset is not limited to 256 colors. But how the hell I´m able to make such tileset when DLTCEP doesn´t allow me to do it?? It works only with 256 colors no matter what. Because I believe glitches are there just due to color reduction, it doesn´t have anything to do with zooming. Just tested it.

Still it would be perfect to have area editor in NI

Oh, one more thing - during conversion of my own custom tilesets strange thing happened - night TIS was converted perfectly (with occasional glitches mentioned above), but day tileset was corrupted - green color which represents transparent sectors was present there, covering all affected tiles, but looked semi transparent somehow, because I was able to see animated water overlay beneath. So I have to export day tileset to PNG image, select all green 0,255,0 color, delete it (means those sectors become transparent), save it back as PNG and then convert it via NI to PVRZ based tileset. Only after then it started to work as intended (with few glitches)...

Acifer

@argent77:
I successfully converted areas with night weds and multiple overlays without any problems and it works fine ingame.
Your tool is a fantastic, must-have aid for all the area creators out there!
'tis a fine day for area editing.


@kiski:
You're right, there are still some lines visible between some water-tiles. The same lines that appear after a correct tis-conversion by hand.
In my case, however, they seem to be linked to zooming in:

Zoomed out (Sorry for the eagle flying by...):

Zoomed in:

They seem to have the size of 1 pixel, so zooming out could just "blur" that lines.

kiski wrote: »

Night TIS was converted perfectly (with occasional glitches mentioned above), but day tileset was corrupted - green color which represents transparent sectors was present there

That's odd. The extended-night-areas were converted correctly using tis2ovl:

Day:

Night:

As you can see, there are again some lines visible in the screenshots, especially in the night-wed.
It could be either related with the "WT" (water-overlay) tiles and not the fault of the area-tis-files at all. Another solution would be it is an EE-engine-glitch, so we have wait for the next patch (...), since the same lines can be seen in some bam-files.
But that's fine for me. There are so many hiccups in that old engine, and that line-problem is one of the smallest we have to face.
The most important thing is that there is now a tool available that reduces the time for the conversion of areas so dramatically you could create Enhanced-Editon-areas "on the fly".

LavaDelVortel wrote: »

is there any thread where you write some more details on what exactly you are working on?

Not yet. I have to admit, it's kind of weird asking for help without talking about what that help is needed for.
I am working on some mods that will add quests and new areas to BG2.
If you're interested, I would be happy if you would take a look at some of my area art on my deviant art page:
https://www.deviantart.com/marthammor
Some screenshots can be found here on my deviant art portfolio:
http://silverrealms.daportfolio.com/gallery/987076
I'll make a proper announcement when the first one of the mods is finished. Now that most of the coding is done, all of the areas are completed and -thanks to argent- everything is "ready to go" in BG2EE, it won't take too long.

kiski

@Acifer Man, now when I see your stuff, I can´t compare to you with my areas Just keep up with an excellent work.
Pity I didn´t save a screenshot from that corrupted day TIS. Now it´s corrected, so I can´t do it. And I also suspect that maybe those water TIS files can do those glitches as well. But funny, in NI during preview I can see it perfectly without errors.

P.S.: Can´t believe I asked you to find some modders... You have already enough work to do yourself

argent77

kiski wrote: »

@argent77 That tool is perfect but I do have glitches... horizontal ones... occasionally, but still. I wonder - PVRZ style tileset is not limited to 256 colors. But how the hell I´m able to make such tileset when DLTCEP doesn´t allow me to do it?? It works only with 256 colors no matter what. Because I believe glitches are there just due to color reduction, it doesn´t have anything to do with zooming. Just tested it.

That's a bug in the renderer of the EE games. It can happen when the tile graphics on the pvrz textures is too fragmented.
I'd suggest to use NI to convert palette-based tileset into the pvrz variant, more specifically the export option provided when you open a tileset from the resource tree, since it takes tileset layout from the associated WED file into account to reduce fragmentation to a minimum. The tileset converter from the "Convert" menu doesn't, which makes it more susceptible to these glitches.

Gwendolyne

Yes. It seems there is nothing to be done until a next patch corrects this behavior, specially with doors overlays.

@Argent77, once again, thanks a lot for your new tool that may save my life when it comes to make my old engine maps compatible with EE games.
However, I did not have time to use it yet. Will it override the old tis file or not? What is the correct command to be used?

TIS2OVL.exe oldmap.tis newmap.tis

TIS2OVL.exe /mymod/tis/oldmap.tis /mymod/newtis

TIS2OVL.exe oldmap.tis

kiski

@Gwendolyne Correct synatx is:

tis2ovl.exe myarea.wed

WED file is addressed, not a TIS one. You should have both WED and asociated TIS in same folder so you don´t have to set path for it... I made my own .bat file whole process.

@argent77 I have already TIS files in pvrz format. Glitches are reduced to minimum, I must admit, but they are still there... Here are the screens, both day and night versions.

Gwendolyne

@kiski Thanks a lot. Once again, it will save my time.

kiski

And I can confirm that glitches are also around the closed door tiles... Almost all of them. Funny only in my own ares, original areas works, more or less... when you open affected door, glitch is gone, because primary tile is visible, secondary is hidden. And when you close it, secondary lay over the primary one and glitch is back

Gwendolyne

kiski wrote: »

And I can confirm that glitches are also around the closed door tiles... Almost all of them. Funny only in my own ares, original areas works, more or less... when you open affected door, glitch is gone, because primary tile is visible, secondary is hidden. And when you close it, secondary lay over the primary one and glitch is back

Yes, EE rendering is very messy when it comes to display outdoor maps with 20 doors... It would be fine if developers might explain how they converted legacy tis files to EE engine. @JuliusBorisov is summoned.

LavaDelVortel

@Acifer I was just curious and those look amazing! I hope to get a chance to give it a run once it's all ready