Flash on iPhone: My Experience

October 10th, 2009 by Ben

Hello, world!

I wanted to post some real world information on Adobe’s latest big announcement, Adobe Flash Applications for iPhone. I participated in the pre-release beta along with several other very talented Flash developers. We got to be some of the first outsiders to work with the iPhone technology. To try out the tech, we wrote and shipped Trading Stuff in Outer Space in just 8 days.

By the way, if this is Flash on iPhone stuff is new to you, you should definitely read Aditya’s “Developing for the Apple iPhone using Flash” article on adobe.com. In addition to being an all around great guy, Aditya was also a huge help in working through the very early beta issues we encountered. Ted Patrick posted some cool iPhone example apps with source. (Not only does Ted make a good jedi, he also helped me debug some rendering issues in Trading Stuff at the last minute. Thanks, Ted!) And Mike Chambers, who brought me into the program, has posted more explanation with a lot of useful links for Flash/iPhone work.

It’s worth stating emphatically that Adobe has a large, talented team working hard on iPhone. I couldn’t hope to properly thank all the Adobe folk who explained, demonstrated, debugged, and implemented through long sleepless nights to build this tech and fix bugs so we could ship Trading Stuff. Guys, you rock!

A brief history of Ben - I spent five years working on 3D and 2D C++ game engine technology before I moved to Flash, so I have a little different background than most Flash developers.

Ok - thanks and context out of the way. Let’s get to the meat. What’s it like to develop for the iPhone with Flash?

First, the toolchain is fantastic. All you do to get on the iPhone is write a Flash app however you like, and then (more or less) click the “compile IPA” button. You get an IPA, you deploy it on your device, and bob’s your uncle. That’s the easy part, and it has been that easy since day one. (For comparison, I’ve seen builds with pre-release and final XBox 360 SDKs, with homebrew PSP and Nintendo DS, with PS3 and Wii - they are all a lot harder to start building on.) Sometimes takes a while for all the optimization to happen, but you don’t need that unless you are packaging a release for distribution.

Deploying to the iPhone is the easy part. The hard part is getting your content to run silky smooth and in an iPhone-friendly way. What do I mean by that? Performance? Sure, you have to be careful with performance, but you also need to make sure you respect the touch interface’s limitations, that you obey Apple’s guidelines, that your art fits the device, and so forth. We spent as much time on that stuff as we did on performance in Trading Stuff.

The main things to watch out for is rendering - you must use hardware acceleration, which acts a lot like cacheAsBitmap - and memory allocation - the iPhone has very slow memory, so the GC running is the kiss of death. I could list a bunch of specifics but at this stage in the game, they would be out of date by the next SDK update. The best thing to do in this and all other optimization cases is to measure, find the biggest bottleneck, change it, and measure again to see if you got a win. Repeat until you have adequate performance.

Louis Gerbarg posted a great analysis of the iPhone tech based on Trading Stuff. Since I wrote Trading Stuff, I figured I should comment on it. :)

On the LLVM piece, Adobe has put in major effort to make this function properly. And it really does work. Even on day one of the prerelease, the compiler worked pretty well. Occasionally you’d hit an optimizer error - but it’s worth noting that in LLVM most optimization operations are separate from the backend. IMHO, Adobe was very smart to use LLVM as opposed to rolling their own or repurposing GCC, due to code maintainability and extensibility issues. You will notice that they have used LLVM before in Alchemy, so there’s prior experience here. And though Louis cites Apple not using LLVM for iPhone compilation as a weakness, Apple is using LLVM for their desktop compilation, which shows that LLVM is going to be a good choice long term.

For the resource bundling, that’s entirely my fault - I chose to embed all my resources in my SWF and when the SWF was converted, it was converted to a binary with those resources in it. I could have put them alongside the SWF and they would have been bundled as part of the IPA (in fact, this is the case for Default.png, which is treated as a normal file even though it has special behavior thanks to OS X). I’m not so sure about this linker behavior Louis is referring to; checking the Mach-O segment reference doesn’t reveal any insights beyond that yes, it is probably bad to keep the compressed version of assets in RAM. But Trading Stuff isn’t paging anyway, making it a non-issue.

For the rendering performance, the build of Trading Stuff that is on the App Store is not using hardware acceleration at all. And the ARM11 is a crappy chip for software rendering. I was showing a build at Adobe MAX that used hardware rendering, and it runs smoothly, like a real game should. I am waiting for a few bugs in Adobe’s SDK to be fixed, at which point I will release a new version on the store. The performance difference is night and day. I’ll post about that when I get the update out.

Let me put a caveat here: the iPhone is something like a tenth of the machine your desktop PC is. So if you want to do incredible 3d graphics and hand-optimize your assembly, maybe you should look into using XCode and Objective C, and build your content and code explicitly for the iPhone. However, if you know Flash (or have a project in it to port) and want to put something out quickly, this is a fantastic path, and worth a look. And it’s only going to improve from this very early version.

Time for some closing thoughts. Adobe has shown that they value the platform and want to bring their content to it by making some serious technology investments. I hope that Apple will respect that, and work with Adobe to make sure Flash works well on the iPhone.

Posted in Opinion, flash |

Viewing 25 Comments

    • ^
    • v
    This is a great write up. I also want to thank you for being a good sport. I chose to use Trading Stuff for my analysis because it is the kind of game I enjoy. I posted an update at the top of my blog post with a link to this.
    • ^
    • v
    No problem. I totally understand the lure of reverse engineering things, and I'm a little flattered you chose my app. :) Thanks for the link on your blog!
    • ^
    • v
    Thanks Ben for highlighting how you have achieved this port, I also got the game because I liked the concept, I'm not criticizing the game concept itself.

    However, I don't get why a bad performance of the actual Flash software renderer is to blame on the "crappy ARM11" chip or the "very slow ram", iPhone is not designed and sold with the "Made For Flash" sticker, Apple give plenty of resources (may I add free?) to develop properly on the iPhone and it is not really necessary to get down to ARM assembler in order get smooth 3D games at high frame rates or any other application.

    It would be worth noting also that the converted Flash app is always HW accelerated as it uses the 3D chip to stamp texture on the screen but it actually uses it in the wrong way. When you use the "cache AsSurface" flag on a sprite you're not activating any special hardware setting in the iPhone side, you're only telling to the Flash soft-renderer to consider an already rendered object as static, in this way, the texture associated to a certain sprite is not rendered again, but for this to work, most if not all display list objects have to be static or very small to avoid the bottleneck in texture transfer from ram to 3D chip.

    Nobody doubt that Adobe teams are full of very talented people, just not sure how focused they are, I bought my CS4 over 1 year ago and I feel kind of frustrated that they jump and down to show CS5 when this stuff that costed me over 2000 euros still crash like it is a beta and nobody seems to worry about fixing it.
    • ^
    • v
    If I can step in here for Ben for a second. People keep talking about HW acceleration as though using any OpenGL call means it is accelerated. Of course they are stamping a texture and putting out on the screen using some HW features. Its not like there is any way to access a physical linear framebuffer and directly blit to it.

    The ARM11 isn't a crappy chip, it is a crappy chip for software compositing, and Ben claims that they will be able to move to HW compositing which gets that out of the critical path. But ultimately, the ARM11 is to blame from the crappy performance of the Flash renderer because it will have bad performance with basically any software renderer. Adobe is to blame for not having a HW renderer ready, but at some point people need to bite the bullet and ship. Don't get me wrong, I'm still skeptical they can pull it off in a generic sense that works well for lots of exisiting Flash code, but the way it works now makes sense from a project management perspective.

    The issue is that the current apps aren't using the GPU to perform any of the compositing. Strictly speaking, that is what one would expect from a fairly new product, for 2 reasons. 1) In most cases you need to have a software renderer in any event because you might fall off the HW path depending on the features you are using. Just look at how an iPhone 3G performs when you use transparencies. 2) It is generally easier to port the software render to new architectures, so it is what is running first, then you write lots of platform specific code to get HW rendering working. This is the same progression as Quartz to Quartz Extreme, etc.

    Also, the ram and software renderer are actually a huge issue. The GPU and CPU share access to the bus and don't necessarily have enough unchip resource to keep multiple frames around. That means that those software composites are reading and writing, causing tons of bus transactions, which then contend with the GPU trying to grab the texture out of ram. I would not be shocking if the frame rate is not bound by CPU, but memory stalls.
    • ^
    • v
    Thanks, Louis - I appreciate your excellent write up. :) (See reply further down for my thoughts on this thread.)
    • ^
    • v
    Didn't post this in my reply lower down but I agree on the CPU thing. Desktop has ~1mb or better cache with prefetch. iPhone has 256kb cache at best, and 16kb at worst, and unclear what prefetch strategy it has. In addition, there is less and slower RAM. Rough estimate puts the iPhone RAM at something like 1/10th the speed of desktop RAM.

    Software rasterization has to have big fast cache and fast RAM to be manageable (that's one big reason why GPU is fast), so even though on MHz it looks like about a 5x gap between desktop 3ghz chip and iPhone CPU, it can actually be much worse depending on workload.

    I know the haters will interpret this as "omg ben is blaming iphone for his performance woes," but they have bad reading comprehension skills. There is a real, measurable difference between desktop and iPhone hardware, and you have to be very careful to get solid performance out of the iPhone.
    • ^
    • v
    I believe most of people reading here comprehend enough the distinction between a portable device and a desktop computer, at least at size level :-) Given that, I understand that someone can define "crappy" the performance of chip only if he/she is comparing a particular device to other devices in the same field rather than to a desktop as it would not make any sense (like saying the beetle has a crappy engine compared to a Ferrari, true, but you're comparing a 10K car to 500K car.....)

    There is at least one company producing a compatible embedded Flash Player using the tessellation method to offload rendering completely to the hardware and even solving the FSAAA problem thanks to it. Their stuff is used in many softwares, honestly I didn't hear anybody laughing about them.
    • ^
    • v
    Are we talking the glut tesselator (http://glprogramming.com/red/chapter11.html) or a custom extension/tesselation routine?
    • ^
    • v
    Since GLUT is not available on OpenGLES, we can only talk about custom tessellation routines.
    • ^
    • v
    Yes, I understand completely what you say. But there is a play on words here, by "HW accelerated" they really mean "not updating the textures so frequently", nothing is really accelerated, they really use the OpenGLES for its ability to blit faster to the screen and nothing else (not completely, the 2.5D stuff actually uses the 3D features but that's a digression.....). This would be equivalent to pre-saving all sprites to PNG from Flash IDE and then load them up as textures (pretty much what we do now) losing anyway the dynamic nature nature of Flash shapes and sprites.

    Instead of rushing to release another version, they really should take an hard look at the code base of the Flash Player in general and start to create several rendering backends, rather just relying on the software one all the time.

    They start to make use of hardware video decoder, but again that's just video, the vector rasterizer is still software; they made use of multicore chip, dividing horizontally the frame buffer by the number of available cores and assign to each core a strip to rasterize. Yes, it is accelerated, by taxing completely the CPU now (look at Mac) and this sounds more like the proverbial use of a cannon to kill mosquitos without really solving the issue at all.

    On the device front, the things are still pretty messed up, I agree that from a project management point of view that was the only (maybe?) option they had, but if the result is so average why bother?

    To efficiently render vector shapes, OpenGL offers some nice features on desktop systems, like the ability to upload polygons and ask to tessellate then texture them directly to the HW, if portable systems could offer similar options it would be almost trivial for Adobe to offer a completely HW based rasterization backend. OpenGLES do not offer such tessellation facility, that means they would have to do it in software (and it is not a trivial thing to do), then upload the resulting triangles strips, probably this will make the things even slower compared to now. Without considering that OpenGLES 1.1 do not even support antialiasing so rasterized polygons would look just horrible.

    More than all the problems above, what surprises me is the closeness of the resulting conversion, I mean, if you know that you cannot target the Flash Player for the iPhone, why just don't take the smart route and allow people to tweak your converted code?

    I have the impression they are selling the illusion to a lot of Flash developer that just pressing a button will do the magic, but targeting Flash Player 10 on 2.5GHz with 4GB RAM is certainly different from a small device.

    And this whole strategy of blaming on the device when it is slow because it cannot run properly a resource hog runtime is plain silly and annoying at best.
    • ^
    • v
    Actually, they are looking at multiple hardware backends - they certainly have them for the other 19 mobile platforms they target. You're right that the current HW support on iPhone is simply better compositing, but that goes a long way - and in fact I, personally, don't even use the vector renderer on the _desktop_, so it's all I need. Most ObjC 3d apps use HW as a fast blitter, too - look at Popcap's games for instance.

    Using the GL tesselator in any serious application is like showing up to a drag race in a VW bug - you're gonna get laughed off the track. One of the big reasons Flash is popular with artists is that it offers 100% consistent quality/appearance across platforms. They don't care that much about speed; they want it to accurately reflect their design down to the pixel. My sense is that if Flash violated that promise, they would lose massive market share amongst artists.

    So they are stuck (to a certain degree) with the software rasterizer. On higher end devices (with better pixel shader support) it is possible to match the software renderer, so as 3d capabilities get better you'll see more of that. But they have to cater to the low-end users to keep their market share, so only when the lowest end users have great penetration with that hardware can they switch wholesale to HW support. If they do it too soon they risk splitting their userbase and making content creation a huge pain.

    I don't think anyone is blaming the device for performance unduly. Look - bottom line, it is a slower device than a PC. Even with hardware acceleration. So getting Flash to run smoothly on it is like putting the proverbial ten pounds of crap in a five pound bag. The same applies for every mobile and set top device. If you want to just do a port that is mediocre, you can do that with little work using the new compiler; if you want to make it good, it's more work (but still a lot less work than porting to ObjC). It is 100% possible to build great content on the iPhone, with Flash or with ObjC or with MonoTouch or Python or iTGB or many other technologies.
    • ^
    • v
    To me, just pressing a button and getting a working iPhone app is pretty much magic? Why not trust Adobe's skilled engineers to make something worthwhile rather than bashing Alpha quality software? ;-)

    I honestly have no illusions about final speed but Flash Authoring targeting other platforms than SWF is a very interesting development! iPhone authoring from a PC is cool by itself, isn't it?

    J
    • ^
    • v
    I've always enjoyed reading your write ups Ben... even though tons of it blows right over my head. But hey, this is the way we learn. Sit at the table with people who know more than you. This appears to provide an awesome toolset and pipeline for getting game ideas onto the iPhone. Looking forward to more of the stuff you guys are pumping out in the "Lab"

    -Benjamin Bradley
    • ^
    • v
    Hey Ben - awesome to know you're a reader. :) All this iPhone stuff is in Flash CS5, so when it's out you can be your very own iPhone developer. :)
    • ^
    • v
    I won't have to wait for CS5, since I've been working with the iPhone SDK for the past few months and I'm aiming to release my first iPhone app in November. Not bad for a bizdev/marketing guy. :)

    And me not waiting for CS5 is probably a good thing for all iPhone owners, I'm sure they don't want me releasing "80's Night with Pat" again.
    • ^
    • v
    i've played native iphone games and they run very well. no lagging, brief load times, few crashes. 'trading stuff' seems extremely laggy.

    your blame of the iphone's hardware sounds exactly like the excuse an adobe developer was using in the adobe forums. he claimed cs4 did not work properly with exposรฉ (window switcher) in os x because apple didn't support their software. the thing is that cs4 apps were the only ones with problems and just so happen to be the ones not following apple's design guidelines. cs4 is a sloppy mess of 'fake' windows floating on top of an actual os x window.
    so just as the adobe developer blamed apple for their own poor design choices, you are blaming apple for not supporting flash's clearly unsupported platform. flash needs so many resources for such simple tasks. what you and adobe are doing are creating a very tall truck and are now asking cities to rebuild their bridges taller to support your truck which should be shorter in the first place.

    i am a flash developer myself, but i keep it on the desktop. i really hope apple continues disregarding flash for the iphone until (and if) adobe fixes its massive performance issues, which are entirely their fault.
    • ^
    • v
    One of my main points was that the performance issues are Adobe's fault in the sense that at the time I needed to put the app on the store for them to demo it, the hw support wasn't in. I am waiting for Adobe's SDK to stabilize a bit before I ship, but I have a version of Trading Stuff running on my phone right now that runs great.

    The iPhone hardware is great. It's a fantastic phone, with good GPU, decent CPU, and great build quality. I'm very happy Adobe is taking advantage of its capabilities, and I want to stress it is entirely possible to build good apps with their tech. I'm sorry you dislike Adobe's UI decisions, but I don't think that that reflects on the quality of their Flash port.
    • ^
    • v
    good post
    • ^
    • v
    This is by far the best blog post and discussion on CS5's Flash>iPhone that I've read yet.
    • ^
    • v
    Thanks, Matt! (For the compliment and the tweet. :)
    • ^
    • v
    "the iPhone is something like a tenth of the machine your desktop PC is. So if you want to do incredible 3d graphics and hand-optimize your assembly, maybe you should look into using XCode and Objective C, and build your content and code explicitly for the iPhone."

    Or you could just use Unity ;) No BS, I've got a project running an 8,000 poly boned character (duck) with ragdoll physics running nicely on iPod Touch 2G and above. Granted iPhone 3G and below hardware runs a bit slow but it's still very playable. The iPhone hardware is excellent, especially for game studios with a content creation bottleneck which wouldn't allow for any more detailed games even with powerful desktop hardware.

    http://www.slidetoplay.com/video/ragduck-hunt

    Otherwise, great article on early Flash iPhone development. I checked Trading Stuff out and thought it was pretty neat, especially for an 8 day project. I'm looking forward to seeing The Incredible Machine and Grunts:Skirmish on the platform.
    • ^
    • v
    Sure, that is also an option. :) You will notice that their actual renderer is running in C on the device, though, so I don't think I was being too misleading. ;)

    Cool game. Kind of gorey. :P

    We'll see what comes out on the iPhone. We're more web oriented than device oriented, but we are always looking for cool projects. :)
    • ^
    • v
    Ben, just to digress slightly from the more technical side of things (and because the support URL for "Trading Stuff" gives me a DNS error ...) are you aware of a bug in "Trading Stuff" where after playing for a while you're unable to buy any more trade goods? The buy side gets this really long red line which extends to the left edge of the screen and you can't buy anything at all at that point. I've never been able to complete a game yet because of this issue ...
    • ^
    • v
    Great write-up. It's nice to see flash finally appearing on the iPhone, albeit in app-form rather than in a fancy plugin. :)

    A pity i haven't really taken flash seriously as a game development platform before this.

    Performance-wise, getting into the "this is an embedded device with slow hardware" mindset does take a bit of getting used to. Nowadays i just refer to my little list of things which i know is slow. e.g. uncompressed texture uploads, disk i/o, various system api's, etc.
    • ^
    • v
    I'm glad to see any article based on actual experience developing Flash for the iPhone. I'd like to see more detail though. Maybe after you finish optimizing for the GPU you could write up some specifics of what you learn along the way?
 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus