HaxeFlixel Tutorial: Mask

Announcement: I’m entering crunch-time for a game project I’m working on. Hopefully it’ll finish by early August. I may not be able to update on the regular weekends, but I will try.

I’m a fan of image manipulation, but I can’t comprehend the math behind it all. In fact, before writing this tutorial, I spent about 4 hours researching and troubleshooting masking in HaxeFlixel.

We’re going to cover image masking today. I apologise in advance if the information below are inaccurate or inefficient. I’d appreciate any corrections or suggestions — just comment below this post and I’ll get back to you.

Introduction

There are two scenarios I could think of, when masking is necessary:

  • Taking an image and masking it with another image
  • Cutting a hole with a mask image (reverse masking)

As such, we shall proceed with the tutorial with the above two methods as our goal.

Setup

Let’s setup our HaxeFlixel project, as usual:

As for placeholder assets, we shall be using the images included in the Flixel Power Tool Test Suite Github page (Note: cloning didn’t work for me, so I had to download the ZIP file instead).

Masking with two images

PhotonStorm’s Power Flixel Tools allows image masking, as demonstrated in the demo page (specifically, the FlxDisplay page). Thankfully, it was ported to HaxeFlixel under the  flixel.util.FlxSpriteUtil  library.

Let’s start off with loading the two images in MenuState.hx:

And if you build with lime test neko , you’ll get this:

Screen Shot 2015-05-24 at 8.01.12 PM

The output is straight-forward — there’s no masking yet. To mask the two images, you need to merge them into a result FlxSprite, and add that to the stage, like this:

The above code produces this result:

Screen Shot 2015-05-24 at 8.04.15 PM

Note two issues with the output:

  • The x/y position of the base  and mask  FlxSprites doesn’t seem to affect the result  FlxSprite. It merges at the origin top-left point (0,0) by default.
  • The base  image’s transparency isn’t preserved after merging (note the pink background for the character)

As of writing, I couldn’t figure out how to easily fix the above issues. If you need a solution for the issues mentioned, my only suggestion for now is to use your favorite image editor (e.g. Photoshop) and create your desired masked/transparent PNG instead.

Reverse masking

This example seems to be most common. Let’s rewrite the existing code in MenuState.hx:

And the result is as follows:

Screen Shot 2015-05-24 at 8.24.32 PM

Note the curtain  is a semi-transparent black rectangle that covers the whole screen. Let’s say we want to cut a hole where the player and dummy is, perhaps we’d do it like this:

But the result doesn’t seem to be as expected:

Screen Shot 2015-05-24 at 8.34.26 PM

There are three problems with the above output:

  • The circle mask has the same issue as mentioned in “Masking with two images” section above — the base image (the curtain) lost its transparency, which results in a solid white circle.
  • The circle is supposed to be transparent, not opaque.
  • The circle mask does not follow the player’s movement.

Luckily, I managed to find a solution for the above mentioned. First, let’s fix the circle mask:

Note the  invertedAlphaMaskFlxSprite  function, which is a little hack based on the existing  FlxSpriteUtil.alphaMaskFlxSprite  function, as explained from the forum page here. The result is as follows:

Screen Shot 2015-05-24 at 9.03.32 PM

Now let’s make the circle follow the player:

But the result is undesired (or perhaps desired, depending on your intention):

licecap-mask

What exactly is happening? After I did some tinkering, here’s the explanation I could come up with:

  • All FlxSprite images are cached, whether it’s created with makeGraphic  or loaded with loadGraphic .
  • When an image is cached, doing a  makeGraphic  (using the same shape and color) or loadGraphic  (using the same image path) will result in the cached image being used, instead of a new image.
  • In the above code, we tried to create a newMask  FlxSprite, then load the _curtain ‘s image data into it. We assumed the _curtain ‘s image is uncut, which is true. However, the _mask ‘s bitmap data has been cached, so the cached image is used, resulting in the _mask  being continuously re-cut and updated instead.

The first solution I would think of, was to do a clone of the bitmapData instead, which means we don’t use the cached image:

Now, if you test the game, it works as expected; The circle mask now follows the player. However, this is actually a bad idea:

Screen Shot 2015-05-24 at 9.35.43 PM

Note the deteriorating FPS (green chart) and increased memory usage (blue chart).

From what I can tell, the newMask  FlxSprite ends up caching the unique _curtain ‘s bitmapData. This means, with every update()  cycle, a new image is created and cached — eventually, memory will run out and the game will freeze.

To fix this issue, we have to dive deeper into the bitmapData — this is where newbies like me start getting uncomfortable for using image-manipulating API:

Now if you run the game, the performance no longer deteriorates:

Screen Shot 2015-05-24 at 9.51.30 PM

And to wrap things up, we could draw more circles to the single mask, which allows for custom-shaped cut-outs, as demonstrated below:

licecap-mask-2

Note the above GIF, where the circles do not end up overlapping one another.

I also found and article that says the code doesn’t work for flash target as noted here. However, I just tested it with lime test flash -web  and it seems to work fine.

This concludes the tutorial on masking. If I discover a solution for the issues mentioned in the “Masking with two images” section above, I’ll update this post.

Addendum: I found an amazing tutorial that may be relevant, and I highly recommend you check it out:

Sight & Light: how to create 2D visibility/shadow effects for your game

5 thoughts on “HaxeFlixel Tutorial: Mask

  1. Bruce says:

    Hi WY,

    I tried following your tutorial but it didn’t work for me. I’m having trouble drawing circles and as a result the program won’t run.

    I know this tutorial is 2.5 years old so maybe something in Flixel has changed along the way? Can you please provide an update or some insight if possible?

    Thanks so much!

    • WY Leong says:

      Hi Bruce!

      I’m sorry to hear you’re having issues with getting your program to run. Indeed, Haxeflixel has changed a lot since my articles were written. I haven’t worked with Haxe for over a year so I can’t give you any insight off the top of my head. I’ve considered updating/rewriting my Haxeflixel tutorials, but I have other priorities in my life now… so I guess that is very unlikely at the moment.

      I’m sorry, and I hope you manage to figure it out eventually! Your best bet for now would be to ask around in the forums: http://forum.haxeflixel.com/

      • Bruce says:

        Hey WY,

        After tinkering for a while I got a workable version running.

        Thanks for the speedy reply and your comments were invaluable! Wish you all the best and thanks again man!

  2. These sound like bugs (especially the alpha mask needing to be multiplied by -1). Are they? Did you open discussions/issues about them?

Leave a Reply to WY Leong Cancel reply

Your email address will not be published. Required fields are marked *