v2019dev0/TestReports/20190412

From Apertis
Jump to: navigation, search


Testing results considerations:

  1. Manual test cases were executed on target Arm, target AMD64, base SDK AMD64 and SDK AMD64 images when applicable.
  2. All automated test cases already integrated into LAVA system were executed on the following images: target Arm, target AMD64, development AMD64, base SDK AMD64 and SDK AMD64.
  3. Failures and partial failures were filed as bugs in Phabricator for further investigation and analysis.

For OSTree tests, please refer to V2019dev0/TestReports/20190412/OSTree.

For tiny-containers tests, please refer to V2019dev0/TestReports/20190412/tiny-containers.


The platform image used for these tests are:

Automated tests

Automated tests results are available at http://lavaphabbridge.apertis.org/report/v2019dev0/20190412.1

Manual tests

ID Description Type Minimal Arm Minimal SabreLite Minimal Arm64 Minimal AMD64 Target Arm Target AMD64 Base SDK AMD64 SDK AMD64 Tested By Notes
Sanity
sanity-check Check that the Apertis system basically works. (manual) Manual PASSED PASSED PASSED PASSED PASSED PASSED PASSED PASSED sagar
Booting
image-bootable Test whether the image is bootable. Manual PASSED PASSED PASSED PASSED PASSED PASSED PASSED PASSED sagar
image-gui-start Test whether the image starts a graphical environment. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
HW Acceleration
x-hw-accelerated Test whether the correct GLES2 render is used. Manual N/A N/A N/A N/A N/A N/A PASSED PASSED sagar
Multitouch
webkit2gtk-mt-touch-events Test implementation of W3C touch events API in WebKit2GTK. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
WebKit
webkit2gtk-actor-tiles Test implementation of actor-based tiled backing store in webkit2GTK. Manual N/A N/A N/A N/A NOT TESTED NOT TESTED NOT TESTED NOT TESTED sagar testcase deprecated
webkit2gtk-event-handling-redesign Test WebKit2Gtk ability to scroll by touch with inertial behaviour after the touch ended. Manual N/A N/A N/A N/A PARTIAL PARTIAL PARTIAL PARTIAL sagar T4243
Webkit2GTK-AC
webkit2gtk-ac-3d-rendering Test that 3D transforms are applied correctly. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
webkit2gtk-ac-animations Test that Webkit2GTK actors created for representing layers animate properly. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
webkit2gtk-ac-scrolling Test that actors created for representing layers scroll properly. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
Meta Data Management
tracker-indexing-mass-storage Check tracker indexes mass storage devices on insertion. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T4409
Networking
bluez-avrcp-volume Test the AVRCP volume up and down commands. Manual N/A N/A N/A N/A PASSED PASSED FAILED FAILED sagar T6015T6017
bluez-phone Test all BlueZ features needing interaction with a phone (Pairing, OBEX, PBAP, AVRCP, A2DP, HSP, HFP, PAN). Manual N/A N/A N/A N/A FAILED PASSED FAILED FAILED sagar T5807T5804
bluez-setup Test setup of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Manual N/A N/A N/A N/A PASSED PASSED FAILED PASSED sagar T5806
connman-pan-tethering Test ConnMan support for PAN connectivity. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5929
connman-pan-network-access Test ConnMan support for bluetooth PAN Network Access Profile. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5930
connman-services Tests ConnMan services. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
connman-usb-tethering Test ConnMan support for USB tethering (connection side). Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5931
ofono-sms-receive Test receiving SMS messages with oFono. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5913
ofono-sms-send Test sending of SMS with oFono. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5913
SDK Development
sdk-ide-build-run-program Creates a simple test project and confirms that compiling and executing work. Manual N/A N/A N/A N/A N/A N/A N/A PASSED sagar
sdk-ade-remote-debugging Checks ade debug remote debugging functionality by connecting to a remote target running gdbserver Manual N/A N/A N/A N/A N/A N/A FAILED FAILED sagar T5932
sdk-dbus-tools-bustle Test that bustle-dbus-monitor shows services on the session bus. Manual N/A N/A N/A N/A N/A N/A FAILED FAILED sagar T5896
sdk-dbus-tools-d-feet Test that D-Feet both shows services on the session bus and that new services appear automatically. Manual N/A N/A N/A N/A N/A N/A FAILED FAILED sagar T6024
sdk-vb-fullscreen Tests that the SDK VM can be fullscreened in VirtualBox Manual N/A N/A N/A N/A N/A N/A FAILED FAILED sagar T5800
Eclipse
eclipse-sysroot-management Test sysroot management and update within Eclipse. It will go through configuring an existing sysroot in Eclipse, and then updating it from the web with the latest v2019dev0. Manual N/A N/A N/A N/A N/A N/A N/A FAILED sagar T5901
eclipse-plugins-install-target Test copying files to the remote device from within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse. Manual N/A N/A N/A N/A N/A N/A N/A FAILED sagar T5901
eclipse-remote-debugging Test remote debugging within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse. Manual N/A N/A N/A N/A N/A N/A N/A FAILED sagar T6001
Contacts
evolution-sync-bluetooth Synchronizing Bluetooth-paired phone contacts to Evolution. Manual N/A N/A N/A N/A FAILED FAILED N/A FAILED sagar T5900
Connectivity
bluez-hfp Test Hands Free Profile (HFP) BlueZ feature. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5905T6018
rfkill-toggle Test that rfkill is functional. Manual N/A N/A N/A N/A PASSED PASSED N/A N/A sagar
telepathy-ring Support making and receiving phone calls in Telepathy manually, with a bluetooth-paired phone. Manual N/A N/A N/A N/A FAILED FAILED N/A FAILED sagar T5913
Contacts
folks-ofono-backend Reading contacts from 3G modem SIM card. Manual N/A N/A N/A N/A FAILED FAILED N/A FAILED sagar T5913
Multimedia
webkit2gtk-gstreamer1.0 Tests html5 video playback with GStreamer 1.0 in WebKit2GTK Manual N/A N/A N/A N/A FAILED PASSED PASSED PASSED sagar T5906
traffic-control-basic Test basic traffic control support. This test does not use tcmmd but it could catch missing support in the kernel or misbehaving traffic control. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5898
video-animation-on-boot Test whether Plymouth is able to display a graphical theme. Manual N/A N/A N/A N/A FAILED FAILED FAILED FAILED sagar T5798
Security
apparmor-bluez Test AppArmor profile for support for SPP profile (bluez-spp) and core functionalities (bluez-core). Manual N/A N/A N/A N/A PASSED PASSED N/A N/A sagar
apparmor-bluez-avrcp-volume Test AppArmor profile for Bluez AVRCP volume up and down commands. Manual N/A N/A N/A N/A FAILED FAILED N/A N/A sagar T5899
apparmor-bluez-setup Test AppArmor profiles for setting up of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Manual N/A N/A N/A N/A PASSED PASSED N/A N/A sagar
apparmor-ofono Test that AppArmor profiles provided in ofono package work without unexpected auditd log entries. Manual N/A N/A N/A N/A FAILED FAILED N/A N/A sagar T5897
iptables-nmap Test the firewall using nmap from another computer. Manual N/A N/A N/A N/A PASSED PASSED PASSED PASSED sagar
Webkit
webkit2gtk-aligned-scroll Test that scrolling is pinned in a given direction when started mostly towards it. Manual N/A N/A N/A N/A NOT TESTED NOT TESTED N/A N/A sagar testcase deprecated
webkit2gtk-lazy-click Test touch adjustment (lazy click) support in WebKit2GTK Manual N/A N/A N/A N/A NOT TESTED NOT TESTED NOT TESTED NOT TESTED sagar testcase deprecated
webkit2gtk-drag-and-drop Test HTML5 drag-and-drop support in WebKit2GTK Manual N/A N/A N/A N/A FAILED FAILED PASSED PASSED sagar T4490
webkit2gtk-contextual-zoom This test case is about zooming parts of a web page to make it easier to read on devices with limited screen area. The scope of the test is the zooming functionality that takes into account the content in the clicked area, other functionality that belongs to a real browser isn't to be tested here. Manual N/A N/A N/A N/A NOT TESTED NOT TESTED NOT TESTED NOT TESTED sagar testcase deprecated
PASSED - All tests passed.
PARTIAL - Tests mostly passed, with some minor problems.
FAILED - One or more tests failed.
NOT TESTED - Test not run due to some issue, like WIP, Disabled, lacking proper hardware.
CRITICAL test case - release blocker
HIGH priority test case - must be tested every week
MEDIUM priority test case - must be tested at least every other week
LOW priority test case - test when possible