Cosc 5/4730

Android media Part 2:Pictures and Video

Emulators and Samsung

• The emulators all have problems, much of the following code has been tested on the actual phones

• Video playback code works, but the video may not always display in the emulators

• android – Video/picture capture does not work as documented.

• Samsung hardware! – Some Samsung devices have many problems. Something

about their hardware… search for the video and samsung for some possible answers.

PLAY MEDIAAndroid android.media

Supported formats• In general Android’s support is consistent with other mobile

phones. • It supports the 3GP (.3gp) and MPEG-4 (.mp4) file formats.

– 3GP is a video standard derived from MPEG-4 specifically for use by mobile devices.

• As far as codecs go, Android supports H.263, a codec designed for low-latency and low-bitrate videoconferencing applications. H.263 video is supported in either MPEG-4 (.mp4) or 3GP (.3gp) files. Android also supports MPEG-4 Simple Profile in 3GP files (.3gp) as well as H.264.

Android

• First method– For greater flexibility you will need to use the mediaPlayer

and a surfaceView (or TextureView API 14+)• MediaPlayer like the audio and use a surfaceView to display the

video.• There are examples in the API demo, plus several of the books.

• The second method uses a VideoView to display.– The VideoView uses a MediaController to start and stop

and provide functions to control the video play back.– With a MediaPlayer like the audio. prepare(), then start()– This is one I’ll cover in this lecture.

VideoView

• VideoView is a View that has video playback capabilities and can be used directly in a layout.

• We can then add controls (play, pause, forward, back, etc) with the MediaController.

ViewView example

• Get the ViewView out of the layoutvv = (VideoView) this.findViewById( R.id.VideoView);• Setup where the file to play isUri videoUri = Uri.parse(Environment.getExternalStorageDirectory().getPath() + "/example.mp4");vv.setVideoURI(videoUri);• play the videovv.start();

Adding media controllers.

• Very simplevv = (VideoView) this.findViewById(R.id.VideoView);vv.setMediaController(new MediaController(this));• Now media controls will show up on the

screen.

Using native media player

• Call for an Intent and send it.Uri data = Uri.parse(VideoFile);intent.setDataAndType(data, "video/mp4");startActivity(intent);• Remember, your app is now in the

background.

Example code

• The VideoPlay example code– This will play a video from the internet.

– If you can uncomment the code to have it play the video from the sdcard, but you will need to copy the file to the sdcard.

DISPLAYING A PICTURE

Displaying Pictures

– See code already covered to how display pictures.• ImageView for example…

TAKING A PICTUREAndroid

Camera vs Camera2 API

• Starting in API 21 (Lollipop) there is a new set of APIs. – There is no backward compatibly either.

• So first we look at Camera v1 APIs• And then Camera2 APIs– These are more flexible and should allow for “filters”

to be added on the fly to image.• Finally, you can just use an intent have the

native camera app take the picture.

CAMERA V1 APISTaking A Picture

What to use

• Android packages• import android.hardware.CameraDevice; • import android.hardware.CameraDevice.CaptureParams;

– Permissions and features • <uses-permission

android:name="android.permission.CAMERA" />– Note, This one will requirement permission checking in API 23+

• This too, if you change the how the camera is functioning.• <uses-feature android:name="android.hardware.camera" />• <uses-feature

android:name="android.hardware.camera.autofocus" />

Taking a picture

• In brief– CameraDevice cameraDevice =

CameraDevice.open()– To preview you need a surfaceHolder then use

setPreviewDisplay(surfaceHolder) and cameraDevice.startPreview()

– Finally use the takePicture(…) to get the picture– release() and close() the CameraDevice to release

it.

“View Finder”• The view finder is implemented via a SurfaceHolder and SurfaceView

– In the layout, the a surfaceView is used.• Example:public class PicCapture extends Activity implements OnClickListener, SurfaceHolder.Callback, Camera.PictureCallback {…cameraView = (SurfaceView) this.findViewById(R.id.CameraView);surfaceHolder = cameraView.getHolder();surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);surfaceHolder.addCallback(this);• To finally take the picture we need all this too.cameraView.setFocusable(true);cameraView.setFocusableInTouchMode(true);cameraView.setClickable(true);cameraView.setOnClickListener(this);

“View Finder” (2)• The code to implement the surfaceHolder can be very simple.• When created open the camera and set the displaysurfaceCreated() { camera = Camera.open(); try { camera.setPreviewDisplay(holder); catch (IOException exception) { camera.release(); } }• Once it’s ready, start the previewsurfaceChanged() { camera.startPreview(); }

“View Finder” (3)

• When we are donesurfaceDestroyed() { camera.stopPreview(); camera.release(); }

Get the Picture

• Using the Camera.PictureCallBack we implement, we can get the data for the picture and decide what to do it with it.

• In its simplest form, which doesn’t nothing with the picture

public void onPictureTaken(byte[] data, Camera camera) {• byte[] data is the picture that was taken• this just restarts the preview.

camera.startPreview(); }

Get the Picture (2)• To take the picture we use the • camera.takePicture method in the onClick method for the SurfaceView public void onClick(View v) { camera.takePicture(null, null, null, this);}• takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw,

Camera.PictureCallback postview , Camera.PictureCallback jpeg) • We only need the last to get the picture and it show on the previous slide.

– shutter• the callback for image capture moment, or null

– raw• the callback for raw (uncompressed) image data, or null

– postview • the callback with postview image data, may be null

– jpeg• the callback for JPEG image data, or null

CAMERA V2 APISTaking A Picture

Camera2 APIs

• Like v1, this breaks up into 2 major pieces– Viewfinder• Connect it to a SurfaceView or TextureView.

– Taking the picture

Camera2 ViewFinder• First find the “id” for the camera to use

– CameraManager manager = (CameraManager) getActivity().getSystemService(Context.CAMERA_SERVICE);

– String cameraId = manager.getCameraIdList()[0];• In this case just use the first one (tends to be the back camera)

• Then is gets complex, because we open the camera via this– manager.openCamera (String cameraId,

CameraDevice.StateCallback callback, Handler handler)• StateCallback is a listener “first” and the handler is this case is likely null.

– The handler is the thread to use, null for main. Another thread is likely where you would apply a filter to the image, before it is displayed.» Note, someone want to figure this out?

Camera2 ViewFinder (2)• The call back is pretty simple, but the complex code is here.CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() { @Override public void onOpened(CameraDevice camera) { mCameraDevice = camera; //We now have the camera, so setup the capture of the current surface. startPreview(); //in a method since it will be called several times. }

@Override public void onDisconnected(CameraDevice camera) { Log.e(TAG, "onDisconnected"); }

@Override public void onError(CameraDevice camera, int error) { Log.e(TAG, "onError"); } };

startPreview• First we need a surface (from the SurfaceView or Textview). We are

using a surfaceViewSurface surface = mHolder.getSurface();

– mHolder is our surfaceview• mPreviewBuilder =

mCameraDevice.createCaptureRequest( CameraDevice.TEMPLATE_PREVIEW);

mPreviewBuilder.addTarget(surface);– Note, if wanted to display it on more then 1 surfaceview, we then add the

rest here. (I think).• mCameraDevice.createCaptureSession(Arrays.asList(surface),

CameraCaptureStateCallback);– The CameraCapture is listed on the next slide.

startPreview(2)• CameraCaptureSession.StateCallback CameraCaptureStateCallback = new CameraCaptureSession.StateCallback() {

@Override public void onConfigured(CameraCaptureSession session) {

mPreviewSession = session; //now setup the update preview. mPreviewBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO); HandlerThread thread = new HandlerThread("CameraPreview"); //this time we need a thread. thread.start(); Handler backgroundHandler = new Handler(thread.getLooper()); mPreviewSession.setRepeatingRequest(mPreviewBuilder.build(), null, backgroundHandler); // Request endlessly repeating capture of images by this session and place it in the surfaceview } @Override public void onConfigureFailed(CameraCaptureSession session) { Toast.makeText(context, "onConfigureFailed", Toast.LENGTH_LONG).show(); } }, null); //null again is for a handler we don’t need.

Now we have preview

• To take a picture, it has a very similar method but we start with the createCaptureSession method.– We need to setup an ImageReader, so we can save

it as JPEG (or other format), another handler, Surfaces, plus another captureBuilder

Taking the picture• First setup some variables we need reader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 1);

– Width and height handled in the code, but not shown here.outputSurfaces = new ArrayList<Surface>(2);outputSurfaces.add(reader.getSurface()); //here the surface is a reader, which will save to a file later on.

– Again add more surfaces here as needed.• Setup our handler threads here.HandlerThread thread = new HandlerThread("CameraPicture"); thread.start();backgroudHandler = new Handler(thread.getLooper());• Readlistner is shown later.reader.setOnImageAvailableListener(readerListener, backgroudHandler); //this is where the save is.

• configure the catureBuilder, which is built in listener later on.captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);captureBuilder.addTarget(reader.getSurface());captureBuilder.set(CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);• Fix Orientation, getJpegOreientation is shown in the code, but not here. The build for capture is in a lister.int deviceorientation = context.getResources().getConfiguration().orientation;captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, getJpegOrientation(characteristics, deviceorientation));

readerListener• A new image is available, ie the capture,

so save file. ImageReader.OnImageAvailableListener readerListener = new ImageReader.OnImageAvailableListener() { @Override public void onImageAvailable(ImageReader reader) { Image image = null; image = reader.acquireLatestImage(); ByteBuffer buffer = image.getPlanes()[0].getBuffer(); byte[] bytes = new byte[buffer.capacity()]; buffer.get(bytes); save(bytes);}

• This is just a helper method to save the file which at this point is just an array of bytes.

private void save(byte[] bytes) throws IOException { OutputStream output = null; try { output = new FileOutputStream(file); output.write(bytes); } finally { if (null != output) { output.close(); } } } };

Taking the picture (2)

• This is the line you call to actually take the picture

mCameraDevice.createCaptureSession( outputSurfaces, mCaptureStateCallback, backgroudHandler);– Like in the preview, we need the stateCallback• The outputSurfaces, and handler were declared 2 slides

back.

Taking the picture (3)• So this CameraCaptureSession.StateCallback doesn’t deal with the preview, instead is readies to save a file. CameraCaptureSession.StateCallback mCaptureStateCallback = new CameraCaptureSession.StateCallback() { @Override public void onConfigured(CameraCaptureSession session) { session.capture(captureBuilder.build(), captureListener, backgroudHandler); }

@Override public void onConfigureFailed(CameraCaptureSession session) { } };• And the last call back CameraCaptureSession.CaptureCallback captureListener = new CameraCaptureSession.CaptureCallback() { @Override public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request, TotalCaptureResult result) { super.onCaptureCompleted(session, request, result); startPreview(); //start the preview again. }};

WITH AN INTENTTaking A Picture

Via an Intent.

• You can also have the default “camera app” take the picture and return it to your app.

• Also doesn’t require any permissions.

Intent intent = new Intent( android.provider.MediaStore.ACTION_IMAGE_CAPTURE);startActivityForResult(intent, myID);• Picture is returned in onActivityResult

Via an Intent (2)

protected void onActivityResult(int requestCode, int resultCode, Intent data) { super.onActivityResult(requestCode, resultCode, data);//get the picture out of the intent and display in an image view. Bitmap bp = (Bitmap) data.getExtras().get("data"); iv.setImageBitmap(bp);}

Android Example Code

• A note for the example code PicCapture– Remember it uses touch to take the picture.– Both a V1 and v2 APIs are used in separate fragments.

• PicCapture2 uses a button and separates the code a little better.– Both a V1 and v2 APIs are used in separate fragments.

• PicCapture3 uses an intent to take the picture.• CameraPreview is only camera2 APIs and it separated up

into multiple classes to make the code easier to read.– Similar to how it shown in the slides.

RECORDING VIDEOAndroid android.media

CAMERA V1 APISRecording VIDEO

First…

• Most examples and code on the web and from the android books, DO NOT work.– lots of subtle errors

• debugging is made more difficult, because– the camera throws errors• CameraInput Recording is not ready … frame dropped.

– the AudioFlinger shows constant buffer overflow warnings.

– And this is when the app is working correctly.

Uses a surfaceView

• Like a taking a picture, we need a view finder which uses a surfaceView.

• Call for the MediaRecorder• and setup the encoding.– both audio and video.

Example SurfaceView• in onCreaterecorder = new MediaRecorder();//setup recorder settings Next Slide SurfaceView cameraView = (SurfaceView) findViewById(R.id.CameraView);holder = cameraView.getHolder();holder.addCallback(this); holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);

• implemented methods.public void surfaceCreated(SurfaceHolder holder) { recorder.setPreviewDisplay(holder.getSurface()); recorder.prepare();}

Example SurfaceView (2) public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { }

public void surfaceDestroyed(SurfaceHolder holder) { if (recording) { recorder.stop(); recording = false; } recorder.release(); finish(); }

Recorder Settings• We need to set the sources for audio and videorecorder.setAudioSource( MediaRecorder.AudioSource.MIC); recorder.setVideoSource(MediaRecorder.VideoSource.DEFAULT);

– MediaRecorder.VideoSource.CAMERA should work as well.• Now we need to setup encoders.

– In android 2.2 we can use profiles instead of setting everything manually.– CamcorderProfile.QUALITY_HIGH or QUALITY_LOW

CamcorderProfile cpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);recorder.setProfile(cpHigh);• And set the output locationrecorder.setOutputFile("/sdcard/videocapture_example.mp4");

Recorder Settings (2)

• Manual settings could look like this:recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP);

• QUALITY_HIGH settings are MP4 file• QUALITY_LOW settings are 3gp file• The manual list is very long, see the android doc’s or

Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets, Chapter 11 for a full list of settings.

Recording

• To recordrecorder.start();• To stoprecorder.stop();– The file should be there at this point.

• Remember when you are donerecorder.release();

Example code• The example code

– need an Sdcard to test code and the file will be located /sdcard/videocapture_example.mp4

• The code uses an extended surfaceView call captureSurface, instead of just a surfaceView– The code is all there, but rearranged from the slides.

• Honesty, the code just didn’t work without an extended SurfaceView

• The code starts up with the viewfinder, touch the screen to start recording, again to stop recording. It will then launch the native media player to replay the video.

CAMERA V2 APISRecording VIDEO

Camera2 APIs.

• Not ready yet.

References• http

://developer.android.com/intl/zh-CN/guide/topics/media/index.html• http://www.brighthub.com/mobile/google-android/articles/43414.aspx

(a difficult example to follow and it’s for 1.6)• Apress - Pro Android Media Developing Graphics, Music, Video, and Rich

Media Apps for Smartphones and Tablets– Chapter 2 for taking pictures, chapter 9 for video playback.

• API examples, com.example.android.apis.media• Camera2

– https://github.com/googlesamples/android-Camera2Video– http://

developer.android.com/reference/android/hardware/camera2/package-summary.html

QA&