Mobile Services | AWS Device Farm

Question 1

What is AWS Device Farm?

General

AWS Device Farm | Mobile Services

Answer 1

AWS Device Farm allows developers to increase application quality, time to market, and customer satisfaction by testing and interacting with real Android and iOS devices in the AWS Cloud. Developers can upload their app and test scripts and run automated tests in parallel across 100s of real devices, getting results, screenshots, video, and performance data in minutes. They can also debug and reproduce customer issues by swiping, gesturing, and interacting with a device through their web browser.

Question 2

Who should use AWS Device Farm and why?

General

AWS Device Farm | Mobile Services

Answer 2

AWS Device Farm is designed for developers, QA teams, and customer support representatives who are building, testing, and supporting mobile apps to increase the quality of their apps. Application quality is increasingly important, and also getting complex due to the number of device models, variations in firmware and OS versions, carrier and manufacturer customizations, and dependencies on remote services and other apps. AWS Device Farm accelerates the development process by executing tests on multiple devices, giving developers, QA and support professionals the ability to perform automated tests and manual tasks like reproducing customer issues, exploratory testing of new functionality, and executing manual test plans. AWS Device Farm also offers significant savings by eliminating the need for internal device labs, lab managers, and automation infrastructure development.

Question 3

What types of apps does AWS Device Farm support?

General

AWS Device Farm | Mobile Services

Answer 3

AWS Device Farm supports native and hybrid Android, iOS, and web apps, and cross-platform apps including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks.

Question 4

Does AWS Device Farm use simulators or emulators?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 4

AWS Device Farm tests are run on real, non-rooted devices. The devices are a mixture of OEM and carrier-branded devices.

Question 5

How do I get started with AWS Device Farm?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 5

Please see our getting started guide.

Question 6

Which browsers does the AWS Device Farm console support?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 6

AWS Device Farm works on Internet Explorer 9 or later and the latest versions of Chrome, Firefox, and Safari.

Question 7

Which browsers are supported for testing web applications?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 7

Your web applications will be tested in Chrome on Android and Safari on iOS.

Question 8

What is the maximum file size for apps and tests?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 8

AWS Device Farm supports files up to 4 GB.

Question 9

Do I need to instrument my app or supply source code?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 9

No instrumentation or source code is required to use the built-in tests. Android apps can be submitted as is. iOS apps should be built with “iOS Device” as the target instead of a simulator.

Question 10

Do you store my app, tests, and other files on your servers? For how long?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 10

Apps and files you upload are automatically removed after 30 days. Test results, screenshots, and videos are stored for 15 months. You can also choose to delete files and results at any time through the AWS Device Farm console or API.

Question 11

How do you clean up devices after my testing is completed?

Setting Up Tests & Remote Access Sessions

AWS Device Farm | Mobile Services

Answer 11

After test execution completes, we perform a series of cleanup tasks on each device, including uninstallation of your app. If we cannot verify uninstallation of your app or any of the other cleanup steps, the device will be removed and will no longer be available.

While we continue to add additional cleanup steps and improve the cleanup process, it is possible for data to persist between sessions in some cases, especially if you make use of the device system outside the context of your app. For this reason, and because AWS Device Farm captures video and logs of activity taking place during your use of each device, it is recommended that you avoid providing or entering sensitive information such as account info (E.g., Google Account, Apple ID), personal information, and other security-sensitive details during your automated test and remote access sessions.

Question 12

Do you modify my app?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 12

On iOS, we replace the embedded provisioning profile with a wildcard profile and resign the app. If you provide it, we will add auxiliary data to the application package before installation so the data will be present in your app’s sandbox. Resigning the iOS app results in the removal of certain entitlements. This includes App Group, Associated Domains, Game Center, HealthKit, HomeKit, Wireless Accessory Configuration, In-App Purchase, Inter-App Audio, Apple Pay, Push Notifications, and VPN Configuration & Control.

On Android, we resign the app. This may break functionality that depends on the app signature, such as the Google Maps Android API. It may also trigger anti-piracy and anti-tamper detection available from products such as DexGuard. For built-in tests, we may modify the manifest to include permissions required to capture and save screenshots.

Question 13

What is AWS Device Farm Direct Device Access?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 13

This feature allows developers to access their private fleet devices via their local host machines the same way they debug with real devices connected directly to the machines. On top of the ability for customers to perform the same tasks they do with real devices on Android Studio and Xcode today. Device Farm direct device access gives user the wide range selection of devices and OS versions. To learn more about private devices, please contact us.

Question 14

Does it support both Android and iOS?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 14

Yes.

Question 15

Do I need to additional software on my machine?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 15

Yes. You can down load the client software from one of the following locations.

Windows: https://s3-us-west-2.amazonaws.com/prod-us-west-2-system-resources/AWSDeviceFarmClient/AWS_Device_Farm.exe

MacOS: https://s3-us-west-2.amazonaws.com/prod-us-west-2-system-resources/AWSDeviceFarmClient/AWS_Device_Farm.dmg

For the full instructions for installation please read this documentation.

Question 16

Can I connect to multiple Android devices from my local machine?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 16

Yes.

Question 17

Does this work for public fleet of devices?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 17

No, at the moment, this feature is available for private fleets on AWS Device Farm. To get more information on private fleets: https://aws.amazon.com/device-farm/pricing/#privateDevices.

Question 18

Can I debug and run on AWS Device Farm devices from Android Studio, Xcode, Chrome Developer Tools, Safari Web inspector?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 18

Yes.

Question 19

What test frameworks are supported for Android devices ?

Setting Up for Direct Device Access for Xcode and Android Studio

AWS Device Farm | Mobile Services

Answer 19

Once you connect to Devices, you can run any framework of your choice.

Question 20

Where can I find more information on this feature?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 20

For the detail setup guide, please click here[insert production link]. To learn more about private devices, please contact us.

Question 21

Which devices are available in AWS Device Farm? How do you select the devices in your fleet?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 21

AWS Device Farm has a large (and growing) selection of Android, iOS, and Fire OS devices. We add popular new devices as they are released by manufacturers. We also add new devices as new OS versions are released. See the list of available devices.

Question 22

Does AWS Device Farm have international devices from markets like Europe, China, and India?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 22

We currently have international devices from India. We use market data and customer feedback to continuously update the fleet. If you would like to see a device that isn’t in our fleet, please let us know.

Question 23

How do I select devices? Can I retest on the same device?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 23

For Automated Testing, devices are selected through a collection called a device pool. Some curated device pools are provided automatically, but you can create your own pools, too. During execution, tests will be run against all devices in the specified pool that are compatible with your application and tests. For Remote Access, you select the desired device based on make, model, carrier variant, and operating system version. You can then optionally upload apps and other data as well as configure other device settings. Device Farm then locates an available device matching your request and displays the device’s display in your browser. You can then interact with the device and capture screenshots and video.

Question 24

Are any apps pre-installed on AWS Device Farm test devices?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 24

Yes, test devices will have a number of apps pre-installed by the device manufacturer or carrier.

Question 25

Are devices able to communicate with other services or systems that are available on the Internet?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 25

Yes. All devices have a WiFi connection with Internet access. If your systems are internal (that is, behind a corporate firewall), you can whitelist the IP range 54.244.50.32-54.244.50.63. All device traffic will come from those IPs.

Question 26

Can I test different carrier connections and conditions?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 26

While you can’t test actual carrier connections, you can simulate connection types and conditions using the network shaping functionality. When scheduling a run, you can select a curated network profile like “3G” or “Lossy LTE,” or you can create your own, controlling parameters like throughput, jitter, and loss. All WiFi traffic from the device will be shaped and manipulated for the duration of your tests according to the profile you choose. You can also simulate dynamic environments by changing network parameters from your test scripts.

Question 27

Can I make phone calls or send SMS from the devices?

Selecting Devices

AWS Device Farm | Mobile Services

Answer 27

No, devices do not have carrier connections and cannot make phone calls or send SMS messages.

Question 28

Can I use the device camera?

Testing Your App

AWS Device Farm | Mobile Services

Answer 28

Yes, you can use the device cameras, both front- and rear-facing. Due to the way the devices are mounted, images and videos may look dark and blurry.

Question 29

I don’t have any automated test scripts yet. What do the built-in tests do?

Testing Your App

AWS Device Farm | Mobile Services

Answer 29

The built-in compatibility test suite allows you to install, uninstall, launch, and run Fuzz on the app.

Question 30

What does Fuzz do?

Testing Your App

AWS Device Farm | Mobile Services

Answer 30

Fuzz will perform fuzz testing on your UI immediately after launch. It streams random user input (touches, swipes, keyboard input) in a rapid fashion to your app. You can configure the number of events, the delay between events, and the seed used to randomize events. Using the same seed across test runs will result in the same sequence of events.

Question 31

I test using an automation framework. Which frameworks do you support?

Testing Your App

AWS Device Farm | Mobile Services

Answer 31

For testing iOS, Android, and FireOS apps, we currently support Appium Java JUnit, Appium Java TestNG, Appium Python, Calabash, Instrumentation (Including JUnit, Espresso, Robotium, and any instrumentation-based tests), UI Automation, UI Automator, and XCTest (Including XCUI and KIF). For more information and updated list, visit our documentation.

Question 32

Which test frameworks do you support for web applications?

Testing Your App

AWS Device Farm | Mobile Services

Answer 32

You can run tests written in Appium Java JUnit, Appium Java TestNG, or Appium Python.

Question 33

Can you add support for a modified framework or one I designed myself?

Testing Your App

AWS Device Farm | Mobile Services

Answer 33

We’re always evaluating frameworks to support. Please contact us.

Question 34

How does AWS Device Farm decide when to take a screenshot during a test?

Testing Your App

AWS Device Farm | Mobile Services

Answer 34

If you use one of the supported automation frameworks, you are in full control and can decide when to take screenshots. Those screenshots are included in your reports automatically.

Question 35

Android: Is Google Play Services available on your devices? Which version is installed?

Testing Your App

AWS Device Farm | Mobile Services

Answer 35

Yes, Google Play Services is installed on devices that support it. The services are updated as new versions become available.

Question 36

Android: Is there a default Google account on the devices?

Testing Your App

AWS Device Farm | Mobile Services

Answer 36

No, devices do not have an active Google account.

Question 37

Does AWS Device Farm support record and playback automation or do I have to write my scripts?

Testing Your App

AWS Device Farm | Mobile Services

Answer 37

AWS Device Farm supports frameworks like Espresso and Robotium that have record and playback scripting tools.

Question 38

iOS: Do I need to add your UDIDs to my provisioning profile?

Testing Your App

AWS Device Farm | Mobile Services

Answer 38

No, AWS Device Farm will automatically replace a provisioning profile and resign your app so it can be deployed on our devices.

Question 39

iOS: My app does not contain debug symbols. Can I supply a dSYM file to AWS Device Farm?

Testing Your App

AWS Device Farm | Mobile Services

Answer 39

No, but you can download the logs and symbolicate the stack traces locally.

Question 40

Android: My app is obfuscated. Can I still test my app on AWS Device Farm?

Testing Your App

AWS Device Farm | Mobile Services

Answer 40

Yes, if you use ProGuard. If you use DexGuard with anti-piracy measures, we are unable to re-sign the app and run tests against it.

Question 41

My app serves ads. Will they be displayed on your devices? Will my ad provider flag this as abuse and ban my account?

Testing Your App

AWS Device Farm | Mobile Services

Answer 41

Although devices have access to the Internet, we make no guarantee that ads will be displayed. We recommend that you remove ads from the builds tested on AWS Device Farm.

Question 42

Can I access the machine hosting the device or access its shell as part of my tests? Can I reach the Internet from it?

Testing Your App

AWS Device Farm | Mobile Services

Answer 42

Yes. If you’re using a client-server framework like Calabash, Appium, or UI Automation, you can access the Internet and execute limited shell commands from the host.

Question 43

I’d like to supply media or other data for my app to consume. How do I do that?

Testing Your App

AWS Device Farm | Mobile Services

Answer 43

You can provide a .zip archive up to 4 GB in size. On Android, it will be extracted to the root of external memory; on iOS, to your app’s sandbox. For Android expansion files (OBB), we will automatically place the file into the location appropriate to the OS version. For more information, see the Developer Guide.

Question 44

My app requires dependencies to test all functionality. Can I install other apps?

Testing Your App

AWS Device Farm | Mobile Services

Answer 44

Yes, you can select multiple apps and the order in which to install them. These dependent apps will be installed before your tests begin.

Question 45

Can I test upgrade flows for my app? How do I install an old version of my app?

Testing Your App

AWS Device Farm | Mobile Services

Answer 45

Yes, in order to test your upgrade flow, you can upload and install an old version of your app before the new version is installed and tested.

Question 46

My app makes use of location services. Can I specify the physical location of the device?

Testing Your App

AWS Device Farm | Mobile Services

Answer 46

Yes, you can supply latitude and longitude coordinates that will be used to override a device’s GPS.

Question 47

Can I run localization tests? How do I change the language of the device?

Testing Your App

AWS Device Farm | Mobile Services

Answer 47

Yes, you can provide a locale (for example, “en_US”) to override the default locale setting on a device.

Question 48

How long does it take before my test starts?

Testing Your App

AWS Device Farm | Mobile Services

Answer 48

Tests are immediately queued for execution and usually start within minutes. If one or more devices are not available, test execution for those devices will remain queued until they become available. Testing on the other devices in your test run will continue.

Question 49

What is the maximum test time allowed?

Testing Your App

AWS Device Farm | Mobile Services

Answer 49

The maximum time allowed is 60 minutes. If you need a longer timeout, please contact us.

Question 50

Does AWS Device Farm provide a way to run tests and get results through an API?

Reviewing Results

AWS Device Farm | Mobile Services

Answer 50

Yes. We have a plug-in for the Jenkins continuous integration environment and a Gradle plugin compatible with Android Studio. AWS Device Farm also provides programmatic support for all console features, including setting up a test and downloading test results through an API. For more information, see the AWS Device Farm API Reference. In addition to the API, you can access AWS Device Farm from the AWS SDKs.

Question 51

What’s in an AWS Device Farm test report?

Reviewing Results

AWS Device Farm | Mobile Services

Answer 51

AWS Device Farm test reports contain pass/fail information, crash reports, test logs, device logs, screenshots, videos, and performance data. Reports include both detailed per-device data and high-level results like the number of occurrences of a given error. Remote Access results contain logs and a video of the session.

Question 52

Which device logs are included in an AWS Device Farm report?

Reviewing Results

AWS Device Farm | Mobile Services

Answer 52

AWS Device Farm reports include complete logcat (Android) and device logs (iOS), as well as logs from the device host and specified test framework.

Question 53

My tests generate and save additional log files. Will I see them in my AWS Device Farm reports?

Pricing

AWS Device Farm | Mobile Services

Answer 53

If you write data to logcat (Android) or the device log (iOS), those log entries will be included in the report. AWS Device Farm does not collect any non-standard logs or other artifacts, although you may transfer files via your test script using the device’s or device host’s Internet connection.

Question 54

How much does AWS Device Farm cost?

Pricing

AWS Device Farm | Mobile Services

Answer 54

Pricing is based on device minutes, which are determined by the duration of tests on each selected device. AWS Device Farm comes with a free trial of 1000 device minutes.* After that, customers are charged $0.17 per device minute. As your testing needs grow, you can opt for an unmetered testing plan, which allows unlimited testing for a flat monthly fee of $250 per device.

Question 55

How does the free trial work?

Pricing

AWS Device Farm | Mobile Services

Answer 55

Your first 1000 device minutes are provided free of charge.* This is a one-time trial and does not renew. Once your trial allocation is depleted, you will be billed at the standard rate of $0.17 per device minute.

Question 56

What is a device minute?

Pricing

AWS Device Farm | Mobile Services

Answer 56

A device minute is the billing unit. Device minutes are a measurement of the time it takes (in minutes) to install, execute, and uninstall your app and tests on every device you have selected for your test run. The unit price is constant regardless of the device, test, or application type. Device minutes are only billed for tests that complete without any device or system errors. Similarly, for Remote Access sessions, device minutes are measured from the time it takes to prepare a device to your specification to completely removing any apps and data you placed on the device.

You can dynamically grow and shrink your usage according to your business needs without any upper limits or up-front commitments.

Question 57

What is the unmetered plan and how do device slots work?

Pricing

AWS Device Farm | Mobile Services

Answer 57

Unmetered plans allow unlimited testing and remote access starting at $250 per month. Unmetered pricing is based on the number of device slots you purchase for each usage type (i.e. automated test or remote access) and device family (i.e. Android or iOS) and are priced at $250 per slot per month. Device slots correspond to concurrency.

For instance, if you purchase ten automated test Android device slots and schedule a run on 100 Android devices, Device Farm will execute your tests on up to ten devices at a time until all tests are completed on your selected devices. Purchasing more slots would enable you to get your results faster. Regardless of how many tests or remote access sessions you have in a month, you are billed at the flat rate of $250 per device slot per month. You can cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot). To learn more, see our docs.

Question 58

What if my testing needs change and I need to add or remove device slots?

Pricing

AWS Device Farm | Mobile Services

Answer 58

You can add device slots at any time and they will be available to you immediately. You can also cancel your subscription for one or more device slots at any time and the cancellation will take effect at your next renewal date (the day of the month that you purchased your first active device slot).

Question 59

If I’m on an unmetered plan, can I still make use of metered billing?

Pricing

AWS Device Farm | Mobile Services

Answer 59

Yes. When creating a run, you can choose to make use of your unmetered device slots or use metered device minutes instead. Because concurrency is not limited on metered billing, this gives you the flexibility of running tests faster than would otherwise happen using your device slots.

Question 60

What is a private device?

Pricing

AWS Device Farm | Mobile Services

Answer 60

A private device is a physical instance of a phone or tablet that is exclusive to your account. Private devices can have custom, static configurations and run custom OS images. Each device is deployed on your behalf and removed at the end of your subscription.

Question 61

How do private device subscriptions work and how are they priced?

Pricing

AWS Device Farm | Mobile Services

Answer 61

Each private device under your account is considered a private device subscription. The monthly subscription price is tiered on the cost of the device and starts at $200/month. After the minimum subscription period, you can choose to cancel your subscription at any time. Please contact us for more information.