18.09/TestReports/20180905

From Apertis
Jump to: navigation, search


Testing results considerations:

  1. Manual test cases were executed on target Arm, target AMD64 and SDK AMD64 images when applicable.
  2. All automated test cases already integrated into LAVA system were executed on the following images: target Arm, target AMD64, development AMD64, and SDK AMD64.
  3. Failures and partial failures were filed as bugs in Phabricator for further investigation and analysis.

For OSTree tests, please refer to 18.09/TestReports/20180905/OSTree.

For tiny-containers tests, please refer to 18.09/TestReports/20180905/tiny-containers.


The platform image used for these tests are:

ID Description Type Minimal Arm Minimal SabreLite Minimal Arm64 Minimal AMD64 Target Arm Target AMD64 Development Arm Development AMD64 SDK AMD64 Tested By Notes
Sanity
sanity-check Check that the Apertis system basically works. (manual) Not Integrated
sanity-check Check that the Apertis system basically works. (automated) Integrated PASSED PASSED PASSED PASSED PASSED PASSED andrewsh [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]
Booting
image-bootable Test whether the image is bootable. Not Integrated
image-gui-start Test whether the image starts a graphical environment. Not Integrated N/A N/A N/A N/A
HW Acceleration
x-hw-accelerated Test whether the correct GLES2 render is used. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
GStreamer
gstreamer1.0-decode Test video and audio decoders and demuxers on a list of media. Integrated PASSED PASSED PASSED andrewsh [14] [15] [16]
Multitouch
webkit2gtk-mt-touch-events Test implementation of W3C touch events API in WebKit2GTK. Not Integrated N/A N/A N/A N/A N/A
WebKit
webkit2gtk-actor-tiles Test implementation of actor-based tiled backing store in webkit2GTK. Not Integrated N/A N/A N/A N/A N/A
webkit2gtk-event-handling-redesign Test WebKit2Gtk ability to scroll by touch with inertial behaviour after the touch ended. Not Integrated N/A N/A N/A N/A N/A
Webkit2GTK-AC
webkit2gtk-ac-3d-rendering Test that 3D transforms are applied correctly. Not Integrated N/A N/A N/A N/A N/A
webkit2gtk-ac-animations Test that Webkit2GTK actors created for representing layers animate properly. Not Integrated N/A N/A N/A N/A N/A
webkit2gtk-ac-scrolling Test that actors created for representing layers scroll properly. Not Integrated N/A N/A N/A N/A N/A
Gettext
gettext-i18n Check that gettext internationalization works. Integrated PASSED PASSED PASSED andrewsh [17] [18] [19]
Meta Data Management
gupnp-services Run a few tests related to service browsing, introspection, etc. Integrated FAILED FAILED PARTIAL FAILED FAILED PARTIAL andrewsh [20] [21] [22] [23] [24] [25]
librest-unit Run the unit tests that ship with the library against the running system. Integrated PASSED PASSED PASSED andrewsh [26] [27] [28]
libsoup-unit Run the unit tests that ship with the library against the running system. Integrated PASSED PASSED PASSED andrewsh [29] [30] [31]
tracker-indexing-local-storage Check tracker indexing of local storage and journal behaviour. Integrated FAILED FAILED PASSED andrewsh [32] [33] [34]
tracker-indexing-mass-storage Check tracker indexes mass storage devices on insertion. Not Integrated N/A N/A N/A N/A N/A
tumbler-thumbnailing Check that all thumbnailing features of tumbler are working properly (large, small, videos, documents, images). Integrated PASSED andrewsh [35]
Grilo
grilo Check grilo metadata notifications, and filesystem browsing. Integrated PASSED PASSED PASSED PASSED PASSED PASSED andrewsh [36] [37] [38] [39] [40] [41]
DBus
check-dbus-services Check that all activatable D-Bus services use systemd, and that they can be started. Integrated PARTIAL PARTIAL PARTIAL PARTIAL PARTIAL PARTIAL andrewsh [42] [43] [44] [45] [46] [47]
dbus-dos-reply-time Check the reply time of a few D-Bus method calls while dbus-daemon is loaded by D-Bus traffic. Integrated PASSED PASSED PASSED andrewsh [48] [49] [50]
Library modules
canterbury Run the automated test for the library: canterbury Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [51] [52]
didcot Run the automated test for the library: didcot Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [53] [54]
frome Run the automated test for the library: frome Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [55] [56]
newport Run the automated test for the library: newport Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [57] [58]
ribchester Run the automated test for the library: ribchester Integrated N/A N/A N/A N/A PARTIAL N/A N/A PARTIAL andrewsh [59] [60]
rhosydd Run the automated test for the library: rhosydd Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [61] [62]
traprain Run the automated test for the library: traprain Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [63] [64]
Networking
bluez-avrcp-volume Test the AVRCP volume up and down commands. Not Integrated N/A N/A N/A N/A N/A
bluez-phone Test all BlueZ features needing interaction with a phone (Pairing, OBEX, PBAP, AVRCP, A2DP, HSP, HFP, PAN). Not Integrated N/A N/A N/A N/A N/A
bluez-setup Test setup of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Not Integrated N/A N/A N/A N/A N/A
connman Runs some automated tests using the ConnMan D-Bus API. It guarantees most of the basic features are working. Integrated PASSED PASSED PASSED PASSED PASSED PASSED andrewsh [65] [66] [67] [68] [69] [70]
connman-new-supplicant Test the wpa_supplicant interface. Integrated PASSED PASSED PASSED andrewsh [71] [72] [73]
connman-pan-tethering Test ConnMan support for PAN connectivity. Not Integrated N/A N/A N/A N/A N/A
connman-pan-network-access Test ConnMan support for bluetooth PAN Network Access Profile. Not Integrated N/A N/A N/A N/A N/A
connman-services Tests ConnMan services. Not Integrated N/A N/A N/A N/A N/A
connman-usb-tethering Test ConnMan support for USB tethering (connection side). Not Integrated N/A N/A N/A N/A N/A
ofono-sms-receive Test receiving SMS messages with oFono. Not Integrated N/A N/A N/A N/A N/A
ofono-sms-send Test sending of SMS with oFono. Not Integrated N/A N/A N/A N/A N/A
Folks
folks-alias-persistence Ensures user-set contact aliases persist. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [74] [75]
folks-eds-compatibility Checks that contacts created within Evolution Data Server (as will happen when phone contacts get synchronized to an Apertis machine) show up in other contact-based programs. Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [76] [77]
folks-metacontacts-linking Ensures linking meta-contacts works properly. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [78] [79]
folks-metacontacts-unlinking Ensures unlinking meta-contacts works properly. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [80] [81]
folks-telepathy-retrieve-contacts Ensures that Folks can correctly retrieve Telepathy contacts. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [82] [83]
SDK Development
sdk-ide-build-run-program Creates a simple test project and confirms that compiling and executing work. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
Apparmor Basic
apparmor-basic-profiles Test basic apparmor profiles. Only parsing and syntax of profiles are tested for now. Integrated PASSED N/A N/A PASSED andrewsh [84] [85]
CGroups
cgroups-resource-control Run cgroup resource control tests Integrated PARTIAL PARTIAL PARTIAL andrewsh [86] [87] [88]
SDK
ade-command Checks basic test coverage of the ade tool. Tests ade command, sub-command and options and arguments. Integrated N/A N/A N/A N/A N/A N/A N/A N/A
sdk-ade-remote-debugging Checks ade debug remote debugging functionality by connecting to a remote target running gdbserver Manual N/A N/A N/A N/A N/A N/A N/A N/A
sdk-performance-tools-gprof-smoke-test Checks basic gprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [89]
sdk-performance-tools-sysprof-smoke-test Checks basic sysprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [90]
sdk-debug-tools-gdb-smoke-test Checks basic gdb functionality by stopping a program at a set point and checking its backtrace matches expectations. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [91]
sdk-debug-tools-strace-smoke-test Checks basic strace functionality by checking its output when run on a simple command line program. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [92]
sdk-debug-tools-valgrind-smoke-test Checks basic valgrind functionality by running a program with and without memory leaks, ensuring expected results from valgrind. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [93]
sdk-dbus-tools-bustle-smoke-test Test that bustle-dbus-monitor shows services on the session bus. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
sdk-dbus-tools-d-feet-smoke-test Test that D-Feet both shows services on the session bus and that new services appear automatically. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
sdk-code-analysis-tools-gcov-smoke-test Checks basic gcov functionality by running a sample program a few different ways and ensuring that a sufficient portion of its code paths are executed Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [94]
sdk-code-analysis-tools-splint-smoke-test Checks basic splint functionality by checking its output when run on a simple source file. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [95]
sdk-vb-fullscreen Tests that the SDK VM can be fullscreened in VirtualBox Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
Eclipse
eclipse-sysroot-management Test sysroot management and update within Eclipse. It will go through configuring an existing sysroot in Eclipse, and then updating it from the web with the latest version. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
eclipse-install-to-target Test copying files to the remote device from within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
eclipse-remote-debugging Test remote debugging within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse. Not Integrated N/A N/A N/A N/A N/A N/A N/A N/A
Contacts
evolution-sync-bluetooth Synchronizing Bluetooth-paired phone contacts to Evolution. Not Integrated N/A N/A N/A N/A N/A
folks-metacontacts-antilinking Ensures anti-linking meta-contacts works properly. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [96] [97]
Connectivity
bluez-hfp Test Hands Free Profile (HFP) BlueZ feature. Not Integrated N/A N/A N/A N/A N/A
rfkill-toggle Test that rfkill is functional. Not Integrated N/A N/A N/A N/A N/A
telepathy-ring Support making and receiving phone calls in Telepathy manually, with a bluetooth-paired phone. Not Integrated N/A N/A N/A N/A N/A
telepathy-ring-automated Support making and receiving phone calls in Telepathy. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [98]
Contacts
folks-ofono-backend Reading contacts from 3G modem SIM card. Not Integrated N/A N/A N/A N/A N/A N/A
folks-search-contacts Ensures that Folks can correctly search contacts. Integrated N/A N/A N/A N/A FAILED N/A N/A FAILED andrewsh [99] [100]
abstract-contact-addressbook-creation A way to be able to create abstract addressbook is needed in SyncEvolution, so that separate addressbooks can be created for each paired device. Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [101] [102]
folks-sync-only Test SYNCHRONIZE_ONLY flag for persona stores in Folks. Integrated N/A N/A N/A N/A N/A N/A N/A N/A PASSED andrewsh [103]
storage-arbitrary-data-contacts Support for storing arbitrary data in contacts in Folks. Integrated N/A N/A N/A N/A PASSED N/A N/A PASSED andrewsh [104] [105]
Multimedia
webkit2gtk-gstreamer1.0 Tests html5 video playback with GStreamer 1.0 in WebKit2GTK Not Integrated N/A N/A N/A N/A N/A
traffic-control-basic Test basic traffic control support. This test does not use tcmmd but it could catch missing support in the kernel or misbehaving traffic control. Not Integrated N/A N/A N/A N/A N/A
video-animation-on-boot Test whether Plymouth is able to display a graphical theme. Not Integrated N/A N/A N/A N/A
Robustness
glib-gio-fs Run the GLib gio tests that do I/O on the local filesystem. Integrated PASSED PASSED PASSED PASSED PASSED PASSED andrewsh [106] [107] [108] [109] [110] [111]
Security
apparmor Functional testing for AppArmor functionalities. Integrated PARTIAL N/A N/A PARTIAL andrewsh [112] [113]
apparmor-chaiwala-system Test that AppArmor's system wide rules works according to Apertis security goals. Integrated PASSED N/A N/A PASSED andrewsh [114] [115]
apparmor-functional-demo Show some main AppArmor functionalities, like transitions, net access or resource permissions, under the form of tests. Integrated PASSED N/A N/A PASSED andrewsh [116] [117]
apparmor-utils Test apparmor_parser and other tools in apparmor package can be executed. Integrated PARTIAL N/A N/A PARTIAL andrewsh [118] [119]
apparmor-session-lockdown-no-deny Test that the session lockdown profile is not blocking more than it should Integrated N/A N/A
apparmor-gstreamer1.0 Test that AppArmor profiles provided in GStreamer 1.0 works without Auditd complaining Integrated FAILED N/A N/A FAILED andrewsh [120] [121]
apparmor-pulseaudio Test that AppArmor profiles provided in PulseAudio 2.1 work without Auditd complaining. Integrated FAILED N/A N/A PARTIAL andrewsh [122] [123]
apparmor-bluez-automated Test AppArmor profile for support for SPP profile (bluez-spp) and core functionalities (bluez-core). Not Integrated N/A N/A N/A N/A N/A N/A
apparmor-bluez-avrcp-volume Test AppArmor profile for Bluez AVRCP volume up and down commands. Not Integrated N/A N/A N/A N/A N/A N/A
apparmor-bluez-setup Test AppArmor profiles for setting up of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Not Integrated N/A N/A N/A N/A N/A N/A
apparmor-folks Test that AppArmor profiles provided in Folks package work without unexpected auditd log entries. Integrated FAILED N/A N/A PARTIAL andrewsh [124] [125]
apparmor-geoclue Test that geoclue apparmor profiles are working properly Integrated PASSED N/A N/A PASSED andrewsh [126] [127]
apparmor-ofono Test that AppArmor profiles provided in ofono package work without unexpected auditd log entries. Not Integrated N/A N/A N/A N/A N/A N/A
apparmor-tracker Tests that the tracker abstraction and tracker profiles don't cause false negatives and that it does not allow arbitrary reading of files in the home directory. Integrated FAILED N/A N/A PASSED andrewsh [128] [129]
apparmor-tumbler Tests that the tumbler profile doesn't cause false negatives and that it does not allow arbitrary reading of files in the home directory. Integrated N/A N/A PASSED andrewsh [130]
iptables-basic Test the firewall automatically. Integrated PASSED PASSED PASSED PASSED PASSED PARTIAL andrewsh [131] [132] [133] [134] [135] [136]
iptables-nmap Test the firewall using nmap from another computer. Not Integrated N/A N/A N/A N/A N/A N/A
apparmor-dbus Functional testing for AppArmor D-Bus mediation. Integrated N/A N/A
System Update
boot-performance Test boot performance. Not Integrated N/A N/A N/A N/A N/A
boot-performance-automated Test boot performance. Integrated PASSED PASSED PASSED PASSED PASSED PASSED andrewsh [137] [138] [139] [140] [141] [142]
Webkit
webkit2gtk-aligned-scroll Test that scrolling is pinned in a given direction when started mostly towards it. Not Integrated N/A N/A N/A N/A N/A N/A
webkit2gtk-lazy-click Test touch adjustment (lazy click) support in WebKit2GTK Not Integrated N/A N/A N/A N/A N/A
webkit2gtk-drag-and-drop Test HTML5 drag-and-drop support in WebKit2GTK Not Integrated N/A N/A N/A N/A N/A
webkit2gtk-contextual-zoom This test case is about zooming parts of a web page to make it easier to read on devices with limited screen area. The scope of the test is the zooming functionality that takes into account the content in the clicked area, other functionality that belongs to a real browser isn't to be tested here. Not Integrated N/A N/A N/A N/A N/A
PASSED - All tests passed.
PARTIAL - Tests mostly passed, with some minor problems.
FAILED - One or more tests failed.
NOT TESTED - Test not run due to some issue, like WIP, Disabled, lacking proper hardware.
CRITICAL test case - release blocker
HIGH priority test case - must be tested every week
MEDIUM priority test case - must be tested at least every other week
LOW priority test case - test when possible
Personal tools
Namespaces

Variants
Actions
Navigation
Tools