Trials and Tribulations with Android Camera2 API11 Sep 2018
I recently had the opportunity to work with the Android Camera2 API on a Xamarin project for a client. I found the API tough to start using initially, so I decided to create a post about using it to help reinforce what I learned. I’m definitely not an expert in Camera2 yet, but this should help others (or future me) understand the basics and save a lot of ramp-up time.
All code for this post is available on GitHub. I recommend downloading a copy to follow along on your own. I will include relevent code samples in this post, but I think it’s helpful to see everything together as well.
Disclaimer: I’ve tried to make this sample as complete and usable as possible, but I definitely could have missed edge cases. You should test against many devices, especially making sure to hit both Samsung and non-Samsung phones as there can be odd camera differences between them. If you find a case I missed I’d love to hear about it so I can update this post. Feel free to tweet me @hofmadresu
Before we get started, I wanted to call out the existing resources I found during my search. Both Xamarin and Google provide sample projects for rear-camera picture and rear-camera video. The links in the table below will bring you to the various projects:
While the resources above are all useful and helped me eventually create the functionality I needed, I found them hard to grok as a first time user of this API. What I plan to do with this post is create my own sample application and explain each area to the extent of my understanding. This app will be slightly more complex than the Xamarin and Google examples in order to match the real-world situation that I needed to handle. The biggest additions are:
Support for both front and rear cameras
Support for both photo and video capture on the same screen
A view to display the photo/video after it is taken
The first two points add complications to the code that are tricky to figure out the first time around. The third point provides an easy way to test that we’re capturing the photo/video correctly. I won’t provide much explanation for the preview screens, but I feel the existing samples are harder to understand without them.
Note: For this sample I’m targeting SDK 21. I’m doing keep permissions code from complicating the example. For a real production app, you’ll need to target a modern version and make sure you handle the required permissions (Camera, Microphone, Storage)
The Capture Layout
The first thing we want to do is create a layout where we’ll allow the user to take photos and videos. This will be a mostly straightforward view with 4 main parts: a reverse camera button, a camera preview, a take picture button, and a record video button. We’ll do this with the following code
The only really interesting thing here is the AutoFitTextureView. This is a custom class that inherits TextureView and gives us the ability to adjust its size to fit our camera’s aspect ratio. This prevents the preview image from appearing squashed or stretched.
After adding this code our view looks like this:
Before we really get going with the camera, there are several supporting classes that we’ll need to use.
The Camera2 API is set up to use Java-style callbacks and listeners, so we need to create classes that implement the required interfaces. I think these are cleanest if they just provide an Action that the main class can use to tie into the callback events. We’ll create 4 classes in this category: CameraStateCallback, CaptureStateSessionCallback, CameraCaptureCallback, and ImageAvailableListener. In addition to those we’ll need one comparison class, which we’ll call CompareSizesByArea.
This callback is used during the camera activation phase of our code. We’ll use this when we ask the Camera2 API to open our camera. The API will call one of 3 methods depending on the results of our open request: OnOpened, OnError, and OnDisconnected.
This callback is used during the Preview Configuration phase of both photo and video portions of our code (separate instances, of course). We’ll use this after requesting a preview capture session. The API will then call either OnConfigured or OnConfigureFailed as needed.
This callback is used during the Preview and Image Capture phases of our code. This will allow us to interact with the camera and image during focus, light balance (for flash), and image capture. We’ll need this when we start our preview and when the user takes a photo. The API will call OnCaptureProgressed and OnCaptureCompleted as needed.
We’ll handle OnCaptureProgressed and OnCaptureCompleted the same way. I’m sure there are reasons to handle them differently, however this is one of the areas of Camera2 I don’t yet fully understand.
This listener is after the photo capture process is complete. We’ll use it at the end of the Take Photo process is complete. The API will call OnImageAvailable when it has finished processing the image, and we will use the results to save our photo.
This class will be used in several calculations to determine the correct image and view sizes for our screen and camera configurations. This was provided by both the Xamarin and Android sample applications.
There are a lot of events and actions we need to handle in order to get our camera working. To get it out of the way, we’ll add many of them now and arrange them as close as possible to the flow of the application. One thing you’ll notice as we go through this app is that there is a lot of code to deal with. Because of that, we’re going to split our MainActivity into partial classes to make our code a little easier to work with.
Most of this is just normal initialization. The one interesting thing we’re setting up for later use is our ‘orientations’ field. This will be used to help determine how our image and video captures are oriented (more on this later).
With all of that out of the way, we can finally get to the fun part!
The Camera Preview
Now it’s time to create our camera preview. There is a lot of code in this section. Below you can see the MainActivity.cs file in its entirety, and we’ll follow that up with descriptions of the interesting parts.
Most of this code is taken from the Xamarin sample, though I’ve adapted some parts to fit my needs. I’ll try to call out any functional changes while describing the code
There’s a lot to unpack here, so let’s get started!
This is where everything gets started. Every time the user opens or restores our activity, we want to start up our camera preview. To do this we need to do 2 things: start a background thread and initialize the preview. For the background thread we just call our StartBackgroundThread method, which we’ll look at in the next section. Before we initialize the preview we need to make sure everything is ready for it. Our camera will need access to the TextureView’s SurfaceTexture. It can take a little time for that to become available, especially on first run, so we check if it’s available. If it is we can just continue our process, otherwise we add a listener to the SurfaceTextureAvailable event
Start / Stop Background Thread
These are a couple of straightforward methods that start and stop an Android background thread. The Camera2 API allows use to pass a thread handler into many of our methods and we want to take advantage of that. This allows the camera actions to run without interfering with the user’s actions. I don’t think it’s strictly necessary to use the camera, but that it improves the user’s experience. Start is called in OnResume and Stop is called in OnPause.
This code is taken directly from the Xamarin sample without alteration
These methods are where our real preview code starts. SetLenseFacing handles a few important things for us. The first thing it does is retrieve the cameraId and characteristics for our desired LensFacing (Back vs. Front). Both of these will be used during other steps of the process, so it’s good to store these into a class field. Next it checks if we already have a preview running for the requested LensFacing. If so nothing needs to be done. If we changed LensFacing or there isn’t an existing preview, it stops any existing preview then configures and opens a camera session.
ForceResetLensFacing is used when we need to restart our preview with the same settings it already had. It switches the currentLensFacing field and calls SetLensFacing with the initial value. This may not always be necessary, but it’s good protection in case the situation comes up.
SetUpCameraOutputs is responsible for configuring our image and preview dimensions as-well-as determining if our requested camera supports flash. There’s a lot going on in this method, so we’re going to break it into smaller pieces.
All of the code in this method are adapted from the Xamarin example. Most changes I made were just stylistic, however I allow the front camera whereas their sample does not
The first thing we do is initialize our ImageReader to the correct size. This is what we will use to actually capture the photo. It’s not realy a part of the preview process, but I found this to be a good place to make sure it’s properly initialized. We create it by finding the largest available JPEG size on the device and create a new instance using those dimensions. We also set the maxImages parameter to 1. This controls how many images the reader can keep active in memory at one time. Since we’re planning to save each image to disk immediately we can save memory by only holding one in memory at a time. ImageReader will throw an exception if we try to take a second picture, so we’ll need to clear out our image after saving to disk.
Next we need to determine our preview size and set its aspect ratio. Android has a strange setup with its cameras which makes this more complicated than it would seem at first glance: the camera lens can be rotated differently relative to the phone on different devices. For example: the camera could be rotated 90° on a Nexus 5 and 270° on a Samsung Galaxy S7 (those are just made up examples, I don’t actually know the orientation of various devices).
The last thing we do in this method is check if our camera supports flash. Fortunately, Android provides characteristics for each camera that we can query to check for things like this.
This method is completely pulled from the Xamarin/Google sample code, and is one of the areas I don’t fully understand. It’s used to set our TextureView’s transform, so I believe it works along-side the aspect ratio to make sure our preview appears without stretching or squashing. Fortunately this code is given to us, so we don’t need to figure it out every time we want to use the camera.
OnOpened is called as the ‘success’ callback of manager.OpenCamera() and gives us access to the CameraDevice. We take this opportunity to set our TextureView’s SurfaceTexture buffer size to match our preview size. Then we start a capture session that will be used for both preview and image capture. The capture session needs access to any surface that will be used, so we pass in our TextureView and ImageReader’s respective surfaces.
This is the final step for displaying our preview. It’s called as the ‘success’ callback of CreateCaptureSession(). Here we handle the last setup actions needed like activating auto-focus (if available), activating flash (if available) and starting a repeating capture request. The repeating request tells Android to continuously read from the camera and display the results on our preview surface.
You may ask “Why do we need flash on our preview?”, and my answer to that is “I don’t really know”. It doesn’t actually turn the flash on during preview, but for some reason it needs to be set here for flash to work when we take the photo.
Note that we’re using the cameraCaptureCallback for this request. This seemed weird to me when I first started using the API, as we’ll use that same callback for taking our picture. From what I’ve discovered, this is used here to work with auto-focus and auto-flash as those happen prior to actually taking a picture.
Whew! That was a lot to get through, but we now have a working preview using our device’s rear camera! If we run the application with all of this in place we’ll see something like this:
The Front Camera
Now that we have the preview working for our back camera, we should get implement our switch camera button to activate the front ‘selfie’ camera. Fortunately for us, all of our work on the back camera translates to the front and makes this addition very easy.
Seems almost too easy, doesn’t it? This is one of those nice areas where prior hard work pays off and makes our life easy. Don’t worry, we get into heavy code again while taking the picture 😎
Before we get to taking and especially displaying a photo, we should probably handle cleaning up our state on pause. If we run the app as-is with Logcat attached, we’ll notice that the log goes haywire when we background the app. This is because we’ve left our camera session running and is not meant to run in the background. To fix this we’ll update our OnPause method to call a new CloseCamera method.
Taking a Photo
Now it’s time to turn our preview into an actual photo! To start with we’re going to add a supporting field and method to our activity, then we’ll get into the real code.
In MainActivity_PhotoCapture we’ll add an enum and field to track the state of our capture process. We’ll use this to deal with the auto focus and auto flash period of photo capture.
We need to deal with sensor orientation when saving our photo (and later when capturing videos), so we’ll create a helper method in MainActivity.
Now it’s time to start taking our picture. All of our work for this portion will be in MainActivity_PhotoCapture since it’s all photo-specific.
We’ll begin by adding a LockFocus method and calling it from our TakePictureButton_Click method. Here we check to see if our camera supports auto-focus. If it does we’ll set an auto-focus trigger to our request and start a new capture, using our cameraCaptureCallback to subscribe to the results. If the camera doesn’t support auto-focus, we can just take the picture.
Next we’ll take care of our auto-focus process by implementing ProcessImageCapture. This is called many times during the capture process, and we need to take different actions based on our state.
The first state we’ll handle is WaitingLock. This is the first state we set in LockFocus, and we use it as a first opportunity to check auto-focus. If the auto-focus state is null then everything’s ready to go and we can call CaptureStillPicture. Otherwise we want to see if the state is FocusLocked or FocusNotLocked. In those cases we’re ready to look at auto-exposure (for flash) as well. If the camera doesn’t support auto exposure or if the state is already converged (ready) we can call CaptureStillPicture(). Otherwise we want to call RunPrecaptureSequence() to tell the device to adjust focus and exposure.
Next we’ll handle the WaitingPrecapture state. Here we’re just waiting until the auto-exposure state is either Precapture or FlashRequired, with a null check just in case. Once that condition is met we just change our state WaitingNonPrecapture to move to the next step.
Last we’ll handle the WaitingNonPrecapture. Here we wait until our auto-exposure state is not Precapture. This means that the device has finished focusing and light balancing and we’re ready to call CaptureStillPicture.
Now we want to implement the RunPrecaptureSequence method we called in ProcessImageCapture. This is a short method that sets an auto-exposure trigger and starts a new capture. Without this trigger our auto-flash would not work.
Now we’re ready to capture the still image in CaptureStillPicture(). Here we’ll create a new capture request using the StillCapture template and add our ImageReader surface as a target. Then we’ll set flash and auto-focus to the same settings we used in the preview. Since this is a new request, we also need to deal with orientation again. It’s a little different this time but we can use the GetOrientation method we created earlier to help us. Finally we stop and abort the preview captures and start a new one with our stillCaptureBuilder.
I’ve also added code to make a shutter click sound at this point. It’s not needed for functionality, but it’s something most users will expect
At this point we’ve done everything needed to capture an image from the camera and store it in memory. The last things to do are save the image to disk and show it to the user. We’ll do both by implementing HandleImageCapture. There isn’t a lot special going on here, we’re just saving the image to disk. There are a couple things worth calling out, though:
We need to make sure we call Close on our image. If we don’t the ImageReader never frees up resources and will throw an exception if the user attempts a second photo (remember that we set maxImages to 1)
We need to call UnlockFocus to restore the preview session. This should be done on the UI thread
We’re going to cheat a little showing the image by just asking Android to open the default photo view application with our file.
Finally we’re going to add an UnlockFocus method. This restarts our preview session and the repeating request. After this is complete the user can again see the camera preview.
And that’s that last thing we need to do for image capture! If you’re not looking for video capture you can stop here and ignore the rest of the post. If you’re sticking around until the end, lets move on to recording a video.
Recording a Video
Recording a video is a much simpler process than taking a picture. We’re going to create a MediaRecorder instance tied to our preview surface and tell Android to use that for the video source.
To start with we’ll create a few class fields and a helper method for our video’s size. The method comes directly from Xamarin’s video sample.
Next we’re going to implement the RecordVideoButton_Click method. We want this to start recording if there isn’t already an ongoing recording and stop if there is one. Most of the work is done in another method called PrepareMediaRecorder and in the videoSessionStateCallback, but there are a couple interesting things here too:
We create a new capture session for the recording using both the previewSurface and mediaRecorder.Surface. This is similar to how we set up the preview session using the imageReader.Surface
mediaRecorder.Stop() can throw an exception if the user records a 0 length video. We don’t deal with that here, but it should be handled in a real app
We always reset the mediaRecorder and close the captureSession when finishing a recording
Now we need to implement PrepareMediaRecorder. This is where we apply all our settings for our video. This is a fairly straightforward process. I think the only interesting portion is that we’re using GetOrientation again to deal with camera rotation.
Finally we’ll implement the OnVideoSessionConfigured method. Here we create a new Preview capture request and set the auto-focus to ContinuousVideo (if available). We then start a repeating request for our new request then tell the mediaRecorder to start recording.
If you made it this far, congratulations! We’re finally done creating a basic app that uses the Android Camera2 API! Hopefully you leave this post with a greater understanding than when you started. There’s still plenty more to learn about Camera2 if you’re interested in going into more depth or playing with other features, but I think this is enough to get started with (and this post is long enough as it is 😄).