Abstract: Robotic systems have subsystems with a combinatorially large configuration space and hundreds or thousands of possible software and hardware configuration options interacting non-trivially. The configurable parameters are set to target specific objectives, but they can cause functional faults when incorrectly configured. Finding the root cause of such faults is challenging due to the exponentially large configuration space and the dependencies between the robot's configuration settings and performance. This paper proposes CaRE - a method for diagnosing the root cause of functional faults through the lens of causality. CaRE abstracts the causal relationships between various configuration options and the robot's performance objectives by learning a causal structure and estimating the causal effects of options on robot performance indicators. We demonstrate CaRE's efficacy by finding the root cause of the observed functional faults and validating the diagnosed root cause by conducting experiments in both physical robots (Husky and Turtlebot 3) and in simulation (Gazebo). Furthermore, we demonstrate that the causal models learned from robots in simulation (e. g., Husky in Gazebo) are transferable to physical robots across different platforms (e. g., Husky and Turtlebot 3).

Abstract: Although tests are traditionally used as mandatory summative assessments at the end of the instruction period and their purpose is measuring how well a student has learned a given subject, they can also play an important role in formative assessment when they are employed as an instrument to aid in the acquisition and retention of knowledge. This study evaluates the role of tests as weekly formative assessments and investigates the impact of their repetition (i.e., administering the same test more than once) on academic performance of students in a postgraduate formal methods course for professional software engineers. Our hypothesis is that repetition in formative assessments leads to increased academic performance, as well as to an improvement in the effectiveness of gauging student progress. Based on an analysis of data from a population of 194 students that took the course during the period 2013-2017, we report results that point at the validity of our hypothesis.

Abstract: This paper presents Fortis, a tool for automated, formal analysis and repair of robust discrete systems. Given a system model, an environment model, and a safety property, the tool can be used to automatically compute robustness as the amount of deviations in the environment under which the system can continue to guarantee the property. In addition, Fortis enables automated repair of a given system to improve its robustness against a set of intolerable deviations through a process called robustification. With these techniques, Fortis enables a new process for developing robust-by-design systems. The paper presents the overall design of Fortis as well as the key details behind the robustness analysis and robustification techniques. The applicability and performance of Fortis are illustrated through experimental results over a set of case study systems, including a radiation therapy system, an electronic voting machine, network protocols, and a transportation fare system.

Abstract: When designing and evolving software architectures, architects need to consider large design spaces of architectural decisions. These decisions tend to impact the quality attributes of the system, such as performance, security, or reliability. Relevant quality attributes might influence each other and usually need to be traded off by architects. When exploring a design space, it is often challenging for architects to understand what tradeoffs exist and how they are connected to architectural decisions. This is particularly problematic in architectural spaces generated by automated optimization tools, as the underlying tradeoffs behind the decisions that they make are unclear. This paper presents an approach to explore quality-attribute tradeoffs via clustering and visualization techniques. The approach allows architects to get an overview of the main tradeoffs and their links to architectural configurations. We evaluated the approach in a think-aloud study with 9 participants from academia and industry. Our findings show that the proposed techniques can be useful in understanding feasible tradeoffs and architectural changes affecting those tradeoffs in large design spaces.

Abstract: The feature interaction problem occurs when two or more independently developed components interact with each other in unanticipated ways, resulting in undesirable system behaviors. Feature interaction problems remain a challenge for emerging domains in cyber-physical systems (CPS), such as the Internet of Things and autonomous drones. Existing techniques for resolving feature interactions take a “winner-takes-all” approach, where one out of the conflicting features is selected as the most desirable one, and the rest are disabled. However, when multiple of the conflicting features fulfill important system requirements, being forced to select one of them can result in an undesirable system outcome. In this paper, we propose a new resolution approach that allows all of the conflicting features to continue to partially fulfill their requirements during the resolution process. In particular, our approach leverages the idea of adaptive requirement weakening, which involves one or more features temporarily weakening their level of performance in order to co-exist with the other features in a consistent manner. Given feature requirements specified in Signal Temporal Logic (STL), we propose an automated method and a runtime architecture for automatically weakening the requirements to resolve a conflict. We demonstrate our approach through case studies on feature interactions in autonomous drones.

Abstract: Modern software systems are deployed in a highly dynamic, uncertain environment. Ideally, a system that is robust should be capable of establishing its most critical requirements even in the presence of possible deviations in the environment. We propose a technique called behavioral robustification, which involves systematically and rigorously improving the robustness of a design against potential deviations. Given behavioral models of a system and its environment, along with a set of user-specified deviations, our robustification method produces a redesign that is capable of satisfying a desired property even when the environment exhibits those deviations. In particular, we describe how the robustification problem can be formulated as a multi- objective optimization problem, where the goal is to restrict the deviating environment from causing a violation of a desired property, while maximizing the amount of existing functionality and minimizing the cost of changes to the original design. We demonstrate the effectiveness of our approach on case studies involving the robustness of an electronic voting machine and safety-critical interfaces.

Abstract: In the literature on hyper-parameter tuning, a number of recent solutions rely on low-fidelity observations (e. g., training with sub-sampled datasets) in order to efficiently identify promising configurations to be then tested via high-fidelity observations (e. g., using the full dataset). Among these, HyperBand is arguably one of the most popular solutions, due to its efficiency and theoretically provable robustness. In this work, we introduce HyperJump, a new approach that builds on HyperBand's robust search strategy and complements it with novel model-based risk analysis techniques that accelerate the search by skipping the evaluation of low risk configurations, i. e., configurations that are likely to be eventually discarded by HyperBand. We evaluate HyperJump on a suite of hyper-parameter optimization problems and show that it provides over one-order of magnitude speed-ups, both in sequential and parallel deployments, on a variety of deep-learning, kernel-based learning and neural architectural search problems when compared to HyperBand and to several state-of-the-art optimizers.

Abstract: To meet the demands of high availability and optimal performance in dynamic environments, modern systems deploy autonomic or self-adaptation mechanisms. However, increasingly today's enterprise systems are compositions of many sub- systems, each an adaptive system. Currently, each autonomic manager operates to maintain locally defined quality-of-service (QoS) objectives, but their independent actions often lead to globally sub-optimal results. Commonly, human admin istrators handle situations in which the collection of autonomic systems is behaving sub-optimally. However, generating a plan to change the configurations of the constituent autonomic managers is a complex and challenging task in the management of a single autonomous system, but the challenge is exacerbated where there may be tradeoffs in how to balance configuration options across the collection of autonomic subsystems. These challenges can be addressed by introducing an automated approach, referred to as meta-management, that provides a formal basis for reasoning about changes to the configurations of autonomic subsystems. The automated approach to meta-management is then established as part of a framework that can be used to instantiate a higher level autonomic manager, referred to as a meta-manager, that provides assurance about, and improves the performance of a collection of autonomic systems. This approach and framework includes a MAPE-K control loop specialized to the needs of meta-management, a domain specific language, SEAM, that enables the practical specification of adaptation policies, and a taxonomy of strategy synthesis techniques. The practicality, effectiveness, and applicability of the approach are then evaluated against three case studies. The first is an AWS Shopping Cart system in which a meta-manager is estab- lished to manage a collection of autonomic system represented by a front end user interface, a middleware services tier, and a database services tier. This case study was selected to evaluate the ability of the meta-manager to improve the homeostatic operations of the collection of autonomic systems on popular architectural pattern, code base, and operations platform that is in wide industrial use. The second is the Google Control plane in which a meta-manager was established to manage a collection of autonomic systems that suffered a significant outage. This case study was selected because it presented a well documented and specific failure scenario that occurred during the period of the research of this thesis that cause of which was, partially, a result of human-centric management of a collection of autonomic systems. Finally, the third is a simulation of an electrical grid cascade failure that represents the Northeast Blackout of 2003. This case study was selected because it presents an example of a failure of human-centric management of a collection of autonomic systems that was exhaustively documented that occurred in a context outside of information technology and/or cloud based providers. This provides credibility to the applicability claim of the thesis.

Abstract: As the size and complexity of modern IT systems increases, there is greater need for automatic recovery from failures. Recently, self-adaptive control loops have started to replace human oversight as means to ensure high availability of software systems. Two critical pieces of the self-adaptive loop for high availability are failure identification and fault localization. Failure identification – figuring out that some- thing is not working – is a challenging activity as (1) the monitoring is not done at the same abstraction level as the failures manifest themselves, and (2) because sys- tems perform several activities concurrently, incorrect behavior will appear mixed with correct behavior. Identifying faults, pinpointing the source of the failure, is also challenging as (1) there may be multiple explanations for a fault and (2) diagnosis must be performed in a useful time frame. In this thesis, we propose to improve self- diagnosis through a framework that allows a system to identify failures and pinpoint the corresponding faulty parts in a running system. This framework is based on two key principles: reasoning about the system’s behavior at the software architecture level and providing a declarative approach to describe system behavior. The use of architectural models allows the diagnostic infrastructure to scale gracefully, supports efficient run-time execution of common fault localization algorithms, and supports failure diagnosis of system-level properties such as end-to-end performance. The use of a declarative approach to behavior allows one to systematically specify rules for bridging the gap between low-level monitoring and higher-level problem detec- tion. It also supports reuse across systems that share a common architectural style or implementation infrastructure.