GPUImage

Here at NSHipster, we’re all about diving into the darker corners of Objective-C to learn something new about the systems we interact with every day. Often, this means sifting through Apple frameworks or language features (it is, after all, a lot of what it means to work in Objective-C). However, on occasion, it’s nice to take a look to the burgeoning landscape of third-party libraries and frameworks (and there are some truly remarkable ones) for a glimpse of what’s new and great outside of Cupertino.

This week, we’ll be taking a look at one of the most impressive open source projects you’ll find: GPUImage. Buckle up, NSHipsters—if you’re not careful, you may well end up creating a camera app by the end of the article.

---

GPUImage is a BSD-licensed iOS library written by Brad Larson that lets you apply GPU-accelerated filters and other effects to images, live camera video, and movies.

GPU vs. CPU

Every iPhone ships with two processors: a CPU, or Central Processing Unit and a GPU, or Graphics Processing Unit. Each processor has its own strengths, and modern chip architecture (like in the iPhone’s A4) integrate the CPU and GPU onto the same physical die.

When you write C or Objective-C code in Xcode, you’re generating instructions that will be handled almost exclusively by the CPU. The GPU, by contrast, is a specialized chip that is especially well-suited for computation that can be split out into many small, independent operations, such as graphics rendering. The kinds of instructions understood by the GPU are quite different from that of the CPU, and as such, we write this code in a different language: OpenGL (or specifically, OpenGL ES on the iPhone & iPad).

Check out Jeff LaMarche’s GLProgram OpenGL ES 2.0 book for a great introduction to OpenGL ES and the rendering pipeline.

Comparing the performance of GPU-based rendering to CPU rendering for something like video, the differences are staggering:

CPU vs. GPU Frame-Rate (Larger FPS is Better)

Calculation	GPU FPS	CPU FPS	Δ
Thresholding ⨉ 1	60.00	4.21	14.3⨉
Thresholding ⨉ 2	33.63	2.36	14.3⨉
Thresholding ⨉ 100	1.45	0.05	28.7⨉

Oh, so it’s like Instagram?

Let me put it this way:

Instagram : GPUImage :: Disposable Camera : NASA Space Optics Manufacturing Center

To put it another way, within GPUImage’s APIs lay thousands of camera apps, just waiting for the right combination of filters and a little spark of imagination.

Here’s a table of the 125 (!) filters that come with GPUImage:

Color Adjustments	Image Processing	Blending Modes	Visual Effects
Brightness Filter Exposure Filter Contrast Filter Saturation Filter Gamma Filter Levels Filter Color Matrix Filter RGB Filter Hue Filter Tone Curve Filter Highlight Shadow Filter Lookup Filter Amatorka Filter Miss Etikate Filter Soft Elegance Filter Color Invert Filter Grayscale Filter Monochrome Filter False Color Filter Haze Filter Sepia Filter Opacity Filter Solid Color Generator Luminance Threshold Filter Adaptive Threshold Filter Average Luminance Threshold Filter Histogram Filter Histogram Generator Average Color Luminosity Chroma Key Filter	Transform Filter Crop Filter Lanczos Resampling Filter Sharpen Filter Unsharp Mask Filter Fast Blur Filter Single Component Fast Blur Filter Gaussian Blur Filter Single Component Gaussian Blur Filter Gaussian Selective Blur Filter Gaussian Blur Position Filter Median Filter Bilateral Filter Tilt Shift Filter Box Blur Filter 3x3 Convolution Filter Sobel Edge Detection Filter Threshold Edge Detection Filter Canny Edge Detection Filter Harris Corner Detection Filter Noble Corner Detection Filter Shi-Tomasi Corner Detection Filter Non Maximum Suppression Filter X/Y Derivative Filter Crosshair Generator Dilation Filter RGB Dilation Filter Erosion Filter RGB Erosion Filter Opening Filter RGB Opening Filter Closing Filter RGB Closing Filter Local Binary Pattern Filter Low Pass Filter High Pass Filter Motion Detector Hough Transform Line Detector Line Generator Motion Blur Filter Zoom Blur Filter	Chroma Key Blend Filter Dissolve Blend Filter Multiply Blend Filter Add Blend Filter Subtract Blend Filter Divide Blend Filter Overlay Blend Filter Darken Blend Filter Lighten Blend Filter Color Burn Blend Filter Color Dodge Blend Filter Screen Blend Filter Exclusion Blend Filter Difference Blend Filter Hard Light Blend Filter Soft Light Blend Filter Alpha Blend Filter Source Over Blend Filter Color Burn Blend Filter Color Dodge Blend Filter Normal Blend Filter Color Blend Filter Hue Blend Filter Saturation Blend Filter Luminosity Blend Filter Linear Burn Blend Filter Poisson Blend Filter Mask Filter	Pixellate Filter Polar Pixellate Filter Polka Dot Filter Halftone Filter Crosshatch Filter Sketch Filter Threshold Sketch Filter Toon Filter Smooth Toon Filter Emboss Filter Posterize Filter Swirl Filter Bulge Distortion Filter Pinch Distortion Filter Stretch Distortion Filter Sphere Refraction Filter Glass Sphere Filter Vignette Filter Kuwahara Filter Kuwahara Radius 3 Filter Perlin Noise Filter CGAColorspace Filter Mosaic Filter JFAVoronoi Filter Voronoi Consumer Filter

Seriously, the Filter Showcase Example App that comes bundled in the repository could easily retail on the AppStore for $3.99, as-is. Add Twitter integration and a few sound effects, and you could bump that up to a respectable$6.99.

Rendering Pipeline

GPUImage is, at its core, an Objective-C abstraction around a rendering pipeline. Source images from the camera, network, or disk are loaded and manipulated according to a chain of filters, and finally outputted either a view, graphics context, or data stream.

For example, images from the video camera could have a Color Levels filter applied to simulate different types of color blindness and displayed in a live view.

GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc]
    initWithSessionPreset:AVCaptureSessionPreset640x480
           cameraPosition:AVCaptureDevicePositionBack];
videoCamera.outputImageOrientation = UIInterfaceOrientationPortrait;

GPUImageFilter *filter = [[GPUImageLevelsFilter alloc] initWithFragmentShaderFromFile:@"CustomShader"];
[filter setRedMin:0.299 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[filter setGreenMin:0.587 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[filter setBlueMin:0.114 gamma:1.0 max:1.0 minOut:0.0 maxOut:1.0];
[videoCamera addTarget:filter];

GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:self.view.bounds];
[filter addTarget:filteredVideoView];
[self.view addSubview:filteredVideoView];

[videoCamera startCameraCapture];

Or, combining various color blending modes, image effects, and adjustments, you could transform still images into something worthy of sharing with your hipster friends (example taken from FilterKit, which is built on GPUImage):

GPUImageFilterGroup *filter = [[GPUImageFilterGroup alloc] init];

GPUImageSaturationFilter *saturationFilter = [[GPUImageSaturationFilter alloc] init];
[saturationFilter setSaturation:0.5];

GPUImageMonochromeFilter *monochromeFilter = [[GPUImageMonochromeFilter alloc] init];
[monochromeFilter setColor:(GPUVector4){0.0f, 0.0f, 1.0f, 1.0f}];
[monochromeFilter setIntensity:0.2];

GPUImageVignetteFilter *vignetteFilter = [[GPUImageVignetteFilter alloc] init];
[vignetteFilter setVignetteEnd:0.7];

GPUImageExposureFilter *exposureFilter = [[GPUImageExposureFilter alloc] init];
[exposureFilter setExposure:0.3];

[filter addGPUFilter:exposureFilter];
[filter addGPUFilter:monochromeFilter];
[filter addGPUFilter:saturationFilter];
[filter addGPUFilter:vignetteFilter];

---

Looking through all of what GPUImage can do, one can’t help but get excited. Easy enough to get started immediately (without needing to know anything about OpenGL) yet performant enough to power whatever you dream up. And not just that, but it also comes with a dizzying number of building blocks—all of the color adjustments, blending modes, and visual effects you could ever want (or never knew you needed).

GPUImage is a rare treat for the open source community, and we as Mac & iOS developers are lucky to have it at our disposal. Use it to make something great, and show others the world in a whole new way.