Here’s what we know Samsung’s Bixby assistant can do on the Galaxy S8

 

Samsung’s new virtual assistant, Bixby, is a different sort of thing than what you’re used to. Where Siri, Google, and Alexa are focused on collecting and understanding information out on the internet and then answering your questions about it, Bixby is trying to help you use your own phone. Samsung’s goal is to make it so that if you don’t want to, you don’t have to touch your device at all — “anything you can do with touch, you can do with voice” is the common refrain we’ve heard from them.

That’s an awfully ambitious goal, though, and it’s nowhere near complete. Right now, Bixby offers a much smaller subset of that goal — it only works with 10 or so of Samsung’s own Android apps — so we’ll need to wait to see how it’s updated over time. On stage, Samsung said that it will work with Google Play Music and other third-party apps — and really emphasized how it will work with Samsung Connect smart home devices.

But the funny thing about Bixby is that it actually does way more than that simplistic pitch. It actually consists of three different components that are all interrelated.

Bixby Voice: talk instead of touch

This is the headline feature for Bixby, and we know that it can work in at least the following Samsung apps: phone, messaging, settings, camera, reminder, contacts, and gallery. In those apps, instead of trying to find the thing you want, you can just ask Bixby to do it for you.

In a demo Samsung showed us, Bixby was able to rotate a photo, play a video, and even send a video over to a television screen. You just hit the Bixby button, ask it to do a thing, and then it tries to do that thing.

There are some cases where Bixby might not be able to fully complete a task. One example might be that you want to send an email, but you need to punch in the address yourself. 

Samsung says you can mix both voice and touch interactions, doing whatever is appropriate to the context of what you’re doing. You can hold the phone up to your ear to have a “more private” conversation with Bixby, or change the gender sound of its voice.

At launch, Bixby will work in US English, and Korean, with Chinese and US Spanish to follow shortly after.

Bixby Vision: Augmented reality (for buying stuff)

The part of Bixby we did get to test a bit was the Vision aspect. It’s essentially an augmented reality camera that can identify objects in real time and then search for them on various services. You can launch it directly in the camera app or from Bixby Home.

Even though it’s not all that dissimilar from stuff we’ve seen before like Google Goggles or Amazon Firefly, Samsung’s implementation seems miles better than the competition. It can translate text, read QR codes, and recognize landmarks. But lots of apps can do those things. Where Bixby excels is combining them all with what seems like pretty decent object recognition right there in your camera app.

To use it, you just point it at a thing and wait a tick (or several ticks, depending on the quality of your internet connection). In our tests it recognized flowers and a watch, and it can also recognize stuff like wine labels and books. Samsung has a few specific partners it’s working with — Amazon, Vivino, and Pinterest come to mind — and more are coming.

It’s probably my favorite thing Bixby does.

Bixby Home: yet another pane of widgets

When you swipe over from your home screen or hit the Bixby button, you’ll get taken to Bixby home. It’s a vertically scrolling list of a bunch of information that Bixby can interact with. You’ll find stuff like weather, fitness activity, buttons for controlling your smart home gadgets, and more. 

Samsung calls it a “social stream for your device,” which sounds nice I suppose, but it also sounds like a lot of other attempts at giving you relevant information on your home screen. It could be super useful or it could be a thing you never bother to look at. Samsung says it will learn from your routine and habits (which is another promise we’ve heard before from other companies, notably Google), so it will all come down to implementation.

More to come

Bixby can do a smattering of a few other things, like setting reminders based on your location, querying Wolfram Alpha, and more. So the distinction between Bixby and the Google Assistant on the S8 is probably going to get fuzzier as time goes on.

Waiting for time to go on is really the main thing we’ll need to so with Bixby. Giving it its very own button before it works with everything you’d want it to is a very bold gamble. If it works out, Samsung will have created a new and interesting user interaction model for its devices, something that will entice users not just to get a pretty phone, but buy into a whole ecosystem. That’s a big if, and to get there Samsung will need to prove it works and prove that people use it.