Abstract: Microservice developers increasingly use resiliency patterns such as Retry and Circuit Breaker to cope with remote services that are likely to fail. However, there is still little research on how the invocation delays typically introduced by those resiliency patterns may impact application performance under varying workloads and failure scenarios. This paper presents a novel approach and benchmark tool for experimentally evaluating the performance impact of existing resiliency patterns in a controlled setting. The main novelty of this approach resides in the ability to declaratively specify and automatically generate multiple testing scenarios involving different resiliency patterns, which one can implement using any programming language and resilience library. The paper illustrates the benefits of the proposed approach and tool by reporting on an experimental study of the performance impact of the Retry and Circuit Breaker resiliency patterns in two mainstream programming languages (C# and Java) using two popular resilience libraries (Polly and Resilience4j), under multiple service workloads and failure rates. Our results show that, under low to moderate failure rates, both resiliency patterns effectively reduce the load over the application's target service with barely any impact on the application's performance. However, as the failure rate increases, both patterns significantly degrade the application's performance, with their effect varying depending on the service's workload and the patterns' programming language and resilience library.

Abstract: Mobile malware is a malicious code specifically designed to target mobile devices to perform multiple types of fraud. The number of attacks reported each day is increasing constantly and is causing an impact not only at the end-user level but also at the network operator level. Malware like FluBot contributes to identity theft and data loss but also enables remote Command & Control (C2) operations, which can instrument infected devices to conduct Distributed Denial of Service (DDoS) attacks. Current mobile device-installed solutions are not effective, as the end user can ignore security warnings or install malicious software. This article designs and evaluates MONDEO-Tactics5G-a multistage botnet detection mechanism that does not require software installation on end-user devices, together with tactics for 5G network operators to manage infected devices. We conducted an evaluation that demonstrates high accuracy in detecting FluBot malware, and in the different adaptation strategies to reduce the risk of DDoS while minimising the impact on the clients' satisfaction by avoiding disrupting established sessions.

Abstract: This paper focuses on the problem of optimizing system utility of Machine-Learning (ML) based systems in the presence of ML mispredictions. This is achieved via the use of self-adaptive systems and through the execution of adaptation tactics, such as model retraining, which operate at the level of individual ML components. To address this problem, we propose a probabilistic modeling framework that reasons about the cost/benefit trade-offs associated with adapting ML components. The key idea of the proposed approach is to decouple the problems of estimating (i) the expected performance improvement after adaptation and (ii) the impact of ML adaptation on overall system utility. We apply the proposed framework to engineer a self-adaptive ML-based fraud-detection system, which we evaluate using a publicly-available, real fraud detection data-set. We initially consider a scenario in which information on model’s quality is immediately available. Next we relax this assumption by integrating (and extending) state-of-the-art techniques for estimating model’s quality in the proposed framework. We show that by predicting the system utility stemming from retraining a ML component, the probabilistic model checker can generate adaptation strategies that are significantly closer to the optimal, as compared against baselines such as periodic or reactive retraining.

Abstract: Building and operating software-intensive systems often involves exploring decision spaces made up of large numbers of variables and complex relations among them. Understanding such spaces is often overwhelming to human decision makers, who have limited capacity to digest large amounts of information, making it difficult to distinguish the forest through the trees. In this article, we report on our experience in which we used dimensionality reduction techniques to enable decision makers in different domains (software architecture, smart manufacturing, automated planning for service robots) to focus on the elements of the decision space that explain most of the quality variation, filtering out noise, and thus reducing cognitive complexity.

Abstract: Neural networks are often overconfident about their pre-dictions, which undermines their reliability and trustworthiness. In this work, we present a novel technique, named Error-Driven Uncertainty Aware Training (EUAT), which aims to enhance the ability of neural models to estimate their uncertainty correctly, namely to be highly uncertain when they output inaccurate predictions and low uncertain when their output is accurate. The EUAT approach oper-ates during the model's training phase by selectively employing two loss functions depending on whether the training examples are cor-rectly or incorrectly predicted by the model. This allows for pursu-ing the twofold goal of i) minimizing model uncertainty for correctly predicted inputs and ii) maximizing uncertainty for mispredicted in-puts, while preserving the model's misprediction rate. We evaluate EUAT using diverse neural models and datasets in the image recognition domains considering both non-adversarial and adversarial settings. The results show that EUAT outperforms existing approaches for uncertainty estimation (including other uncertainty-aware training techniques, calibration, ensembles, and DEUP) by providing uncertainty estimates that not only have higher quality when evaluated via statistical metrics (e. g., correlation with residuals) but also when employed to build binary classifiers that decide whether the model's output can be trusted or not and under distributional data shifts.

Abstract: The Robot Operating System (ROS) is a highly popular framework and ecosystem that allows developers to build robot software systems from reusable, off-the-shelf components. Systems are often built by customizing and connecting components into a working ensemble via configuration files. While reusable components allow developers to quickly prototype working robots in theory, in practice, ensuring that those components are configured and connected appropriately is fundamentally challenging, as evidenced by the large number of questions asked on developer forums. Developers must be careful to abide by the assumptions of individual components, which are often unchecked and unstated when building their systems. Failure to do so can result in misconfigurations that are only discovered once the robot is deployed in the field, at which point errors may lead to unpredictable and dangerous behavior. Despite misconfigurations having been studied in the broader context of software engineering, robotics software (and ROS in particular) poses domain-specific challenges with potentially disastrous consequences. To understand and improve the reliability of ROS projects, it is critical to identify the types of misconfigurations developers face. To that end, we study ROS Answers, a Q&A platform, to identify and categorize misconfigurations during ROS development. We then conduct a literature review to assess the coverage of these misconfigurations using existing detection techniques. In total, we find 12 high-level categories and 50 sub-categories of misconfigurations. Of these categories, 27 are not covered by existing techniques. To conclude, we discuss how to tackle those misconfigurations in future work.

Abstract: Cyber-physical systems (CPS) with reinforcement learning (RL)-based controllers are increasingly being deployed in complex phys- ical environments such as autonomous vehicles, the Internet-of-Things (IoT), and smart cities. An important property of a CPS is tolerance; i.e., its ability to function safely under possible disturbances and un- certainties in the actual operation. In this paper, we introduce a new, expressive notion of tolerance that describes how well a controller is ca- pable of satisfying a desired system requirement, specified using Signal Temporal Logic (STL), under possible deviations in the system. Based on this definition, we propose a novel analysis problem, called the tol- erance falsification problem, which involves finding small deviations that result in a violation of the given requirement. We present a novel, two- layer simulation-based analysis framework and a novel search heuristic for finding small tolerance violations. To evaluate our approach, we con- struct a set of benchmark problems where system parameters can be configured to represent different types of uncertainties and disturbances in the system. Our evaluation shows that our falsification approach and heuristic can effectively find small tolerance violations.

Abstract: The emergence of powerful automated design tools in many domains is changing the nature of design, as human-intensive activities can be increasingly off-loaded to those tools. Rather than having a human consider only handful of options, as has been done historically, such tools now enable the generation of a large space of potential designs, exhibiting different tradeoffs among competing qualities of merit, and supporting systematic exploration of the design space. At the same time, this paradigm raises new challenges centered on enabling humans to effectively navigate that generated space in order to select a design that best meets their requirements. In this paper we describe our experience in the domain of generative manufacturing, in which we developed a novel design environment for airplane parts manufacturing that incorporates a number of sophisticated design tools and attempts to tackle the emergent problems of design space exploration that are faced by designers of those parts. We use this experience to highlight the challenges that we faced and reflect on their applicability more generally to tool-assisted software design environments.

Abstract: Cyber-physical systems (CPS) are subject to environmental uncertainties such as adverse operating conditions, malicious attacks, and hardware degradation. These uncertainties may lead to failures that put the system in a sub-optimal or unsafe state. Systems that are resilient to such uncertainties rely on two types of operations: (1) graceful degradation, for ensuring that the system maintains an acceptable level of safety during unexpected environmental conditions and (2) recovery, to facilitate the resumption of normal system functions. Typically, mechanisms for degradation and recovery are developed independently from each other, and later integrated into a system, requiring the designer to develop an additional, ad-hoc logic for activating and coordinating between the two operations.

In this paper, we propose a self-adaptation approach for improving system resiliency through automated triggering and coordination of graceful degradation and recovery. The key idea behind our approach is to treat degradation and recovery as requirement-driven adaptation tasks: Degradation can be thought of as temporarily weakening original (i.e., ideal) system requirements to be achieved by the system, and recovery as strengthening the weakened requirements when the environment returns within an expected operating boundary. Furthermore, by treating weakening and strengthening as dual operations, we argue that a single requirement-based adaptation method is sufficient to enable coordination between degradation and recovery. Given system requirements specified in signal temporal logic (STL), we propose a run-time adaptation framework that performs degradation and recovery in response to environmental changes. We describe a prototype implementation of our framework and demonstrate the feasibility of the proposed approach using a case study in unmanned underwater vehicles.

Abstract: Designing a software architecture that satisfies a set of quality-attribute requirements has traditionally been a challenging activity for human architects, as it involves the exploration and assessment of alternative design decisions. The development of automated optimization tools for the architecture domain has opened new opportunities, because these tools are able to explore a large space of alternatives, and thus extend the architect's capabilities. In this context, however, architects need to efficiently navigate through a large space and understand the main relations between design decisions and feasible quality-attribute tradeoffs in a maze of possible alternatives. Although Machine Learning (ML) techniques can help to reduce the complexity of the task by sifting through the data generated by the tools, the standard techniques often fall short because they cannot offer architectural insights or relevant answers to the architect's questions. In this paper, and based on previous experiences, we argue that ML techniques should be adapted to the architecture domain, and propose a conceptual framework towards that goal. Furthermore, we show how the framework can be instantiated by adapting clustering techniques to answer architectural questions regarding a client-server design space.

Abstract: Robotics systems are complex, safety-critical systems that can con- sist of hundreds of software components that interact with each other dynamically during run time. Software components of robot- ics systems often exhibit reactive, periodic, and state-dependent behavior. Incorrect component composition can lead to unexpected behavior, such as components passively waiting for initiation mes- sages that never arrive. Model-based software analysis is a common technique to identify incorrect behavioral composition by checking desired properties of given behavioral models that are based on component state machines. However, writing state machine models for hundreds of software components manually is a labor-intensive process. This motivates work on automated model inference. In this paper, we present an approach to infer behavioral models for systems based on the Robot Operating System (ROS) using static analysis by exploiting assumptions about the usage of the ROS API and ecosystem. Our approach is based on searching for common behavioral patterns that ROS developers use for implementing reac- tive, periodic, and state-dependent behavior using the ROS frame- work API. We evaluate our approach and our tool ROSInfer on five complex real-world ROS systems with a total of 534 components. For this purpose we manually created 155 models of components from the source code to be used as a ground truth and available data set for other researchers. ROSInfer can infer causal triggers for 87 % of component architectural behaviors in the 534 components.

Abstract: Abstract Software systems are designed and implemented with assumptions about the environment. However, once a system is deployed, the actual environment may deviate from its expected behavior, potentially leading to violations of desired properties. Ideally, a system should be robust to continue establishing its most critical requirements even in the presence of possible deviations in the environment. To enable systematic design of robust systems against environmental deviations, this work proposes a rigorous behavioral notion of robustness for software systems. Then, it presents a technique called behavioral robustification, which involves two tactics to systematically and rigorously improve the robustness of a system design against potential deviations.

Specifically, the robustness of a system is defined as the largest set of deviating environmental behaviors under which the system is capable of guaranteeing a desired property. Then, we present an approach to compute robustness based on this definition. On the other hand, the system is not robust against an environment when the environment exhibits deviations causing a violation of the desired property. The robustification method finds a redesign that is capable of satisfying the property under such a deviated environment. In particular, two tactics, namely robustification-by-control and robustification-by-specification-weakening, are introduced. The robustification-by-control tactic formulates the robustification problem as a multi-objective optimization problem with the goal of guaranteeing the desired property while maximally preserving the existing functionality and minimizing the cost of changes to the original design. Then, the specification-weakening tactic is used alongside the control tactic, which allows weakening the property to generate more feasible redesigns that retain more functionality or have a lower cost.

The proposed robustness computation and robustification method are implemented in a tool named Fortis. The applicability and efficiency of these approaches are evaluated through experimental results across five case studies, including a radiation therapy machine, an electronic voting machine, network protocols, a transportation fare system, and an infusion pump machine.

Abstract: Machine learning (ML) models are now commonly used as components in systems. Due to their black-box and data-driven nature, ML components can produce erroneous outputs (in the form of mispredictions) that may critically impact the system's quality of service. Such mispredictions may be caused by component changes, environment changes, or due to ML components' inherent uncertainty and inaccuracy. In the face of these changes, and to cope with mispredictions, self-adaptation arises as a natural solution: systems that monitor and adapt themselves at run time to optimize their system utility. This thesis provides a repertoire of ML adaptation tactics, and a frameworkthat generates policies specifying when to apply each tactic to adapt ML components such that overall system utility is optimized. The development of this framework raises two main challenges: (i) estimating the expected costs and benefits due to the execution of an adaptation tactic; (ii) evaluating the impact of the improved ML predictions on overall system utility. To address the first problem we build predictors that learn to estimate the expected benefits of each adaptation tactic. To solve the second problem we leverage probabilistic model checking methods and instantiate a formal model of the system, capturing the key dynamics of ML components and their impact on expected system utility. The techniques proposed in this thesis are evaluated via two use-cases: credit card fraud detection and machine translation systems. We show that: the self-adaptive ML-based systems built leveraging the proposed framework achieve better system utility than that achievable when employing simpler baselines to guide adaptation, such as periodic or reactive adaptation schemes; the proposed framework is suitable for run-time adaptation in non-critical domains; the framework can be extended to account for multiple adaptation tactics; and that it can be leveraged to plan for the long term when to adapt ML models.