Using the MvvmCross PictureChooser plug-in with a Windows Phone 8.1 XAML app

by Ishai Hachlili 30. January 2015 21:23

As we're getting closer to the release of Windows 10 for both phones and PCs I figured it's time to migrate some of my apps from WP Silverlight to WP XAML.

Most of my apps use MvvmCross, which allows me to use MVVM (and binding) with Xamarin.iOS and Xamarin.Android as well as Windows Phone of course.

 

One of the plugins I often use in my apps is the PictureChooser plugin. This plugin makes it easy to take a picture or choose a picture from the gallery on all platforms with a couple of lines of code in your ViewModel.

Here's a sample ViewModel Code:

        private readonly IMvxPictureChooserTask _pictureChooserTask;

        private const int MaxPicturePixelDimension = 640;
        private const int PictureQuality = 100;

        public FirstViewModel(IMvxPictureChooserTask pictureChooserTask)
        {
            _pictureChooserTask = pictureChooserTask;
        }

        private byte[] _pictureBytes;
        public byte[] PictureBytes
        {
            get { return _pictureBytes; }
            set { _pictureBytes = value; RaisePropertyChanged(()=>PictureBytes); }
        }

        public ICommand SelectPictureCommand
        {
            get { return new MvxCommand(SelectPicture); }
        }

        private async void SelectPicture()
        {
            var pictureStream = await _pictureChooserTask.ChoosePictureFromLibraryAsync(MaxPicturePixelDimension, PictureQuality);

            var memoryStream = new MemoryStream();
            await pictureStream.CopyToAsync(memoryStream);
            PictureBytes = memoryStream.ToArray();
        }
		

The actual code to launch the picture gallery is that one line of code calling ChoosePictureFromLibrary on the PictureChooserTask. The rest of the code deals with displaying the selected picture.

The plugin also includes the TakePictureAsync method, but in Windows Phone 8.1 there is no separate TakePictureTask, instead, the gallery includes a camera button.


The PictureChooserTask worked great in Windows Phone 8, Android and iOS. With a Windows Phone 8.1 XAML app nothing was happening after I selected a picture.

As I started looking into the MvvmCross code to figure it out, I saw this comments:

            // This method requires that PickFileContinuation is implemented within the Windows Phone project and that
            // the ContinueFileOpenPicker method on the View invokes (via the ViewModel) the ContinueFileOpenPicker method
            // on this instance of the MvxPictureChooserTask


So here's what you need to actually do

  1. Add the ContinuationManager class from Microsoft's documentation to your Windows Phone app
  2. In App.xaml.cs add a ContinuationManager property
    public ContinuationManager ContinuationManager { get; private set; }
  3. Also in App.xaml.cs, override OnActivated and add the following two lines:
      ContinuationManager = new ContinuationManager();
      ContinuationManager.Continue(args as IContinuationActivatedEventArgs);
  4. In your ViewModel class add a public method
      public void ContinueFileOpenPicker(object args)
      {
           _pictureChooserTask.ContinueFileOpenPicker(args);
      }
  5. In the code behind for the view, add the interface IFileOpenPickerContinuable and implement the ContinueFileOpenPicker method:
    ((FirstViewModel)ViewModel).ContinueFileOpenPicker(args);

This code should complete the loop. The PictureChooser plugin's ContinueFileOpenPicker will process the image just like it does for other platforms (in my sample ViewModel above, execution will continue with the stream in the SelectPicture method, as expected)

 

 Here's the complete project on GitHub

Tags: ,

MvvmCross

A Quick Tip for UI design in Windows Phone 8

by Ishai Hachlili 30. October 2012 07:45

If you’ve worked on WP7 apps, you probably came across the MetroGridHelper from Jeff Wilcox. If not, what this little helper does is add a grid of squares that helps you align controls in your app’s pages.

If you’re one of those who ignore the commented sections in newly created pages, you might miss that this feature is now included in the default WP8 project. To use it, all you have to do is uncomment the Image tag at the end of the page

<!--Uncomment to see an alignment grid to help ensure your controls are
aligned on common boundaries. The image has a top margin of -32px to
account for the System Tray. Set this to 0 (or remove the margin altogether)
if the System Tray is hidden.

Before shipping remove this XAML and the image itself.-->
<!--<Image Source="/Assets/AlignmentGrid.png" VerticalAlignment="Top" Height="800" Width="480" Margin="0,-32,0,0" Grid.Row="0" Grid.RowSpan="2" IsHitTestVisible="False" />-->

Tags: , ,

Design | Windows Phone 8

Using Text To Speech and Speech Recognition in Windows Phone 8

by Ishai Hachlili 29. October 2012 12:55

With Windows Phone 8 Microsoft added an API for speech recognition and synthesis (TTS).
The combination of these APIs allows you to create conversations with the user, asking them for input with TTS and listening for their replies.

To use this feature you need to add the following capabilities to your app (in the WMAppManifest.xml file)
ID_CAP_SPEECH_RECOGNITION
ID_CAP_MICROPHONE
ID_CAP_NETWORKING
(like all speech recognition solutions, the actual processing is done on the server side, the phone just streams the audio to the server and gets the text result back. that’s why networking must be enabled  for the app)

Recognizing Speech

Here’s a very simple piece of code that will show the speech recognition UI to the user:

private SpeechRecognizerUI _recoWithUI;

private async void SimpleRecognition()
{
//initialize the recognizer
_recoWithUI = new SpeechRecognizerUI();

//show the recognizer UI (and prompt the user for speech input)
var recoResult = await _recoWithUI.RecognizeWithUIAsync();
}

And here’s what a result looks like when using the default grammar

ResultStatus: Succeeded
RecognitionResult: {
RuleName: ""
Semantics: null
Text: "Recognize this text."
TextConfidence: High
Details: {
ConfidenceScore: 0.8237646
RuleStack: COM Object
}
}

(this is the object hierarchy, I’m showing it as a json objects for clarity)

TextConfidence can have the following values: Rejected, Low, Medium, and High. You use this value to figure out how close the returned text is to what the user actually said. If you want the actual score, you can use ConfidenceScore.

Text is the recognized text. it will be empty if the TextConfidence is Rejected.

RuleName is the name of the custom grammar used for this recognition (since I didn’t use one, the value is empty here)

Semantics are related to SRGS grammars (an XML file that defines a more complex grammar you want to use). I will not get into this more advanced option in this post. 

Prompting the user and adding a custom grammar

The longer the user’s input is the harder it is to get accurate recognition. It makes sense to design your application in such a way where the user only needs to give you short answers.

In the following code snippet I’m showing the user a text prompt asking a specific question and I’m showing the possible answers.
I’m also adding the possible answers as a programmatic list grammar. This is the simples way of adding a grammar and it will increase the accuracy and force the API to match only the words I added in the grammar.
This means that if the user said Lace or Ace it will still be recognized as Race.

I’m also disabling the Readout. If you’re creating a speech conversation with your user, the readout might get tedious and slow the user down. repeating every recognition with “Heard you say: “ and the recognized text takes too long and will get annoying. You might want to enable it and certain situations but I would leave it off by default.
You can also hide the confirmation prompt which displays the same text (Heard you say…) by setting Settings.ShowConfirmation to false.

private SpeechRecognizerUI _recoWithUI;

private async void SpeechRecognition()
{
//initialize the recognizer
_recoWithUI = new SpeechRecognizerUI();

var huntTypes = new[]{"Race", "Explore"};
_recoWithUI.Recognizer.Grammars.AddGrammarFromList("huntTypes", huntTypes);

//prompt the user
_recoWithUI.Settings.ListenText = "What type of hunt do you want to play?";

//show the possible answers in the example text
_recoWithUI.Settings.ExampleText = @"Say 'Race' or 'Explore'";

//disable the readout of recognized text
_recoWithUI.Settings.ReadoutEnabled = false;

//show the recognizer UI (and prompt the user for speech input)
var recoResult = await _recoWithUI.RecognizeWithUIAsync();
}

 

Adding a spoken prompt

To create a conversation, you might want to add a spoken text before prompting the user for their response.
You can do that by simple adding the following two lines before the call to RecognizeWithUIAsync

var synth = new SpeechSynthesizer();
await synth.SpeakTextAsync("What type of hunt do you want to play?");

Notice that I’m waiting for the text to be spoken before showing the recognizer UI. This means the current screen will still be visible while the text is spoken and as soon as it ends the recognizer UI will show up and the prompt sound will be played. If you don’t include the await on that line, the text will be playing while the recognizer is already listening.

A better solution would’ve been to include a TTS option for the prompt in the recognizer. I couldn’t find such an option.
Another way to solve this is to create your own UI and use SpeechRecognizer and RecognizeAsync with your own UI.

Here’s a quick code sample:

private async void RecognizeWithNoUI()
{
var recognizer = new SpeechRecognizer();
var huntTypes = new[]{"Race", "Explore"};
recognizer.Grammars.AddGrammarFromList("huntTypes", huntTypes);

var synth = new SpeechSynthesizer();
await synth.SpeakTextAsync("Do you want to play a Race, or, Explore, hunt?");

var result = await recognizer.RecognizeAsync();
}

The main difference is that there’s no automatic handling of failed recognitions. The RecognizeWithUIAsync call tells the user “Sorry, didn’t get that” and asks them to speak again, with the no UI option, you need to handle that yourself using the TextConfidence value.

 

As you can see, it’s very easy and straightforward to add speech recognition and synthesis to your app. Combined with Voice Commands you can create an experience that lets the user launch and control your app without touching their phone. If you’re using voice commands you can start this experience on that target page only, so when the user launches the app with a voice command you will prompt them with TTS and get their replies with speech and when they launch their app from the apps list or tile it will show a normal user interface.

Tags: , , ,

Speech Recognition | Windows Phone 8

About Me

Ishai Hachlili is a web and mobile application developer.

Currently working on Play The Hunt and The Next Line


Recent Tweets

Twitter October 23, 05:22
@BenThePCGuy a standard where that doesn't matter is better. One more reason to get the #Lumia920, wireless charging, no need for microUSB

Twitter October 23, 05:21
@ManMadeMoon where they dance around the issues and don't really talk about them

Twitter October 23, 05:20
@BenThePCGuy are you a @wpdev ?

Twitter October 23, 04:17
@JonahLupton But if it's black it's usually better

Twitter October 23, 02:58
@jongalloway next time ask your 5 year old how to spell

@EShy