Overview¶

Introduction¶

The objective of this Reference Architecture (RA) is to develop a usable Kubernetes-based platform for the Telco industry. The RA will be based on the standard Kubernetes platform wherever possible. This Reference Architecture for Kubernetes will describe the high-level system components and their interactions, taking the goals and requirements from the Cloud Infrastructure Reference Model [243] (RM) and mapping them to Kubernetes (and related) components. This document needs to be sufficiently detailed and robust such that it can be used to guide the production deployment of Kubernetes within an operator, whilst being flexible enough to evolve with and remain aligned with the wider Kubernetes ecosystem outside of Telco.

To set this in context, it makes sense to start with the high-level definition and understanding of Kubernetes. Kubernetes [244] is a “portable, extensible, open-source platform for managing containerised workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available” [245]. Kubernetes is developed as an open source project in the kubernetes k8srepo repository of GitHub.

To assist with the goal of creating a reference architecture that will support Telco workloads, but at the same time leverage the work that already has been completed in the Kubernetes community, RA2 will take an “RA2 Razor” approach to build the foundation. This can be explained along the lines of “if something is useful for non-Telco workloads, we will not include it only for Telco workloads”. For example, start the Reference Architecture from a vanilla Kubernetes (say, v1.16) feature set, then provide clear evidence that a functional requirement cannot be met by that system (say, multi-NIC support), only then the RA would add the least invasive, Kubernetes-community aligned extension (say, Multus) to fill the gap. If there are still gaps that cannot be filled by standard Kubernetes community technologies or extensions then the RA will concisely document the requirement in the Introduction to Gaps, Innovation, and Development chapter of this document and approach the relevant project maintainers with a request to add this functionality into the feature set.

The Kubernetes Reference Architecture will be used to determine a Kubernetes Reference Implementation. The Kubernetes Reference Implementation would then also be used to test and validate the supportability and compatibility with Kubernetes-based Network Function workloads, and lifecycle management of Kubernetes clusters, of interest to the Anuket community. The intention is to expand as much of the existing test frameworks to be used for the verification and conformance testing of Kubernetes-based workloads, and Kubernetes cluster lifecycle management.

Required component versions¶

Component	Required version(s)
Kubernetes	1.26

Principles¶

Architectural principles¶

This Reference Architecture conforms with the Anuket principles:

Open-source preference: for building Cloud Infrastructure solutions, components and tools, using open-source technology.
Open APIs: to enable interoperability, component substitution, and minimise integration efforts.
Separation of concerns: to promote lifecycle independence of different architectural layers and modules (e.g., disaggregation of software from hardware).
Automated lifecycle management: to minimise the end-to-end lifecycle costs, maintenance downtime (target zero downtime), and errors resulting from manual processes.
Automated scalability: of workloads to minimise costs and operational impacts.
Automated closed loop assurance: for fault resolution, simplification, and cost reduction of cloud operations.
Cloud nativeness: to optimise the utilisation of resources and enable operational efficiencies.
Security compliance: to ensure the architecture follows the industry best security practices and is at all levels compliant to relevant security regulations.
Resilience and Availability: to withstand Single Point of Failure.

Cloud Native Principles¶

For the purposes of this document, the CNCF TOC’s (Technical Oversight Committee) definition of Cloud Native applies:

CNCF Cloud Native Definition v1.0 Approved by TOC: 2018-06-11

“Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone.”

The CNCF TUG (Telecom User Group), formed in June 2019, published a set of Cloud Native Principles suited to the requirements of the Telecom community cloudnativeprinciples. There are many similarities with the CNCF principles, briefly that infrastructure needs to be:

scalable
dynamic environments
service meshes
microservices
immutable infrastructure
declarative APIs
loosely coupled
resilient
manageable
observable
robust automation
high-impact changes frequently and predictably

Exceptions¶

Anuket specifications define certain policies and general principles and strive to .. add general principles from common coalesce the industry towards conformant Cloud Infrastructure technologies and configurations. With the currently available technology options, incompatibilities, performance, and operator constraints (including costs), these policies and principles may not always be achievable and, thus, require an exception process. These policies describe how to handle non-conforming technologies. .. add policies:anuket project policies for managing non-conforming technologies from common In general, non-conformance with policies is handled through a set of exceptions. .. add gov/chapters/chapter09:exception types

The following sub-sections list the exceptions to the principles of Anuket specifications and shall be updated whenever technology choices, versions and requirements change. The Exceptions have an associated period of validity and this period shall include time for transitioning.

Technology Exceptions¶

The list of Technology Exceptions will be updated or removed when alternative technologies, aligned with the principles of Anuket specifications, develop and mature.

Table 113 Technology Exceptions¶
Ref	Name	Description	Valid Until	Rationale	Implication
ra2.exc.tec.001	SR-IOV	This exception allows workloads to use SR-IOV over PCI-PassThrough technology.	TBD	Emulation of virtual devices for each virtual machine creates an I/O bottleneck resulting in poor performance and limits the number of virtual machines a physical server can support. SR-IOV implements virtual devices in hardware, and by avoiding the use of a switch, near maximal performance can be achieved. For containerisation the downsides of creating dependencies on hardware is reduced as Kubernetes nodes are either physical, or if virtual have no need to “live migrate” as a VNF VM might.

Scope¶

The scope of this particular Reference Architecture can be described as follows (the capabilities themselves will be listed and described in subsequent chapters):

Kubernetes platform capabilities required to conform to the Reference Model requirements
Support for CNFs that consist wholly of containers
Support for CNFs that consist partly of containers and partly of VMs, both of which will be orchestrated by Kubernetes
Kubernetes Cluster lifecycle management: including Cluster creation/upgrade/scaling/deletion, and node customisation due to workload requirements.

The following items are considered out of scope:

Kubernetes-based Application / CNF Management: this is an application layer capability that is out of scope of Anuket.

Figure 74 Kubernetes Reference Architecture scope¶

Approach¶

The approach taken in this Reference Architecture is to start with a basic Kubernetes architecture, based on the community distribution, and then add detail and additional features/extensions as is required to meet the requirements of the Reference Model and the functional and non-functional requirements of common cloud native network functions.

This document starts with a description of interfaces and capabilities requirements (the “what”) before providing guidance on “how” those elements are deployed, through specifications. The details of how the elements will be used together are documented in full detail in the Reference Implementation.

Definitions¶

Table 114 Definitions¶
Term	Description
Abstraction	Process of removing concrete, fine-grained or lower-level details or attributes or common properties in the study of systems to focus attention on topics of greater importance or general concepts. It can be the result of decoupling.
Anuket	A LFN open-source project developing open reference infrastructure models, architectures, tools, and programs.
CaaS Containers as a Service	A Platform suitable to host and run Containerised workloads, such as Kubernetes. Instances of CaaS Platforms are known as CaaS Clusters.
CaaS Manager	A management plane function that manages the lifecycle (instantiation, scaling, healing, etc.) of one or more CaaS instances, including communication with VIM for master/node lifecycle management.
Cloud Infrastructure	A generic term covering NFVI, IaaS and CaaS capabilities - essentially the infrastructure on which a Workload can be executed. NFVI, IaaS and CaaS layers can be built on top of each other. In case of CaaS some cloud infrastructure features (e.g.: HW management or multitenancy) are implemented by using an underlying IaaS* layer.
Cloud Infrastructure Hardware Profile	Defines the behaviour, capabilities, configuration, and metrics provided by a cloud infrastructure hardware layer resources available for the workloads.
Cloud Infrastructure Profile	The combination of the Cloud Infrastructure Software Profile and the Cloud Infrastructure Hardware Profile that defines the capabilities and configuration of the Cloud Infrastructure resources available for the workloads.
Cloud Infrastructure Software Profile	Defines the behaviour, capabilities and metrics provided by a Cloud Infrastructure Software Layer on resources available for the workloads.
Cloud Native Network Function (CNF)	A cloud native network function (CNF) is a cloud native application that implements network functionality. A CNF consists of one or more microservices. All layers of a CNF are developed using Cloud Native Principles including immutable infrastructure, declarative APIs, and a “repeatable deployment process”. This definition is derived from the Cloud Native Thinking for Telecommunications Whitepaper, which also includes further detail and examples.
Compute Node	An abstract definition of a server. A compute node can refer to a set of hardware and software that support the VMs or Containers running on it.
Container	A lightweight and portable executable image that contains software and all its dependencies. OCI defines Container as “An environment for executing processes with configurable isolation and resource limitations. For example, namespaces, resource limits, and mounts are all part of the container environment.” A Container provides operating-system-level virtualisation by abstracting the “user space”. One big difference between Containers and VMs is that unlike VMs, where each VM is self-contained with all the operating systems components are within the VM package, containers “share” the host system’s kernel with other containers.
Container Engine	Software components used to create, destroy, and manage containers on top of an operating system.
Container Image	Stored instance of a container that holds a set of software needed to run an application.
Container Runtime	The software that is responsible for running containers. It reads the configuration files for a Container from a directory structure, uses that information to create a container, launches a process inside the container, and performs other lifecycle actions.
Core (physical)	An independent computer processing unit that can independently execute CPU instructions and is integrated with other cores on a multiprocessor (chip, integrated circuit die). Please note that the multiprocessor chip is also referred to as a CPU that is placed in a socket of a computer motherboard.
CPU Type	A classification of CPUs by features needed for the execution of computer programs; for example, instruction sets, cache size, number of cores.
Decoupling, Loose Coupling	Loosely coupled system is one in which each of its components has, or makes use of, little or no knowledge of the implementation details of other separate components. Loose coupling is the opposite of tight coupling
Encapsulation	Restriction of direct access to some of an object’s components.
External Network	External networks provide network connectivity for a cloud infrastructure tenant to resources outside of the tenant space.
Fluentd	An open-source data collector for unified logging layer, which allows data collection and consumption for better use and understanding of data. Fluentd is a CNCF graduated project.
Functest	An open-source project part of Anuket LFN project. It addresses functional testing with a collection of state-of-the-art virtual infrastructure test suites, including automatic VNF testing.
Hardware resources	Compute, storage and network hardware resources on which the cloud infrastructure platform software, virtual machines and containers run on.
Host Profile	Is another term for a Cloud Infrastructure Hardware Profile.
Huge pages	Physical memory is partitioned and accessed using the basic page unit (in Linux default size of 4 KB). Huge pages, typically 2MB and 1GB size, allows large amounts of memory to be utilised with reduced overhead. In an NFV environment, huge pages are critical to support large memory pool allocation for data packet buffers. This results in fewer Translation Lookaside Buffers (TLB) lookups, which reduces the virtual to physical pages’ address translations. Without huge pages enabled high TLB miss rates would occur thereby degrading performance.
Hypervisor	A software that abstracts and isolates workloads with their own operating systems from the underlying physical resources. Also known as a virtual machine monitor (VMM).
Instance	Is a virtual compute resource, in a known state such as running or suspended, that can be used like a physical server. It can be used to specify VM Instance or Container Instance.
Kibana	An open-source data visualisation system.
Kubernetes	An open-source system for automating deployment, scaling, and management of containerised applications.
Kubernetes Cluster	A set of machines, called nodes (either workers or control plane), that run containerised applications managed by Kubernetes.
Kubernetes Control Plane	The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers.
Kubernetes Node	A node is a worker machine in Kubernetes. A worker node may be a VM or physical host, depending on the cluster. It has local daemons or services necessary to run Pods and is managed by the control plane.
Kubernetes Service	An abstract way to expose an application running on a set of Pods as a Kubernetes network service.
Monitoring (Capability)	Monitoring capabilities are used for the passive observation of workload-specific traffic traversing the Cloud Infrastructure. Note, as with all capabilities, Monitoring may be unavailable or intentionally disabled for security reasons in a given cloud infrastructure instance.
Multi-tenancy	Feature where physical, virtual or service resources are allocated in such a way that multiple tenants and their computations and data are isolated from and inaccessible by each other.
Network Function (NF)	Functional block or application that has well-defined external interfaces and well-defined functional behaviour. Within NFV, a Network Function is implemented in a form of Virtualised NF (VNF) or a Cloud Native NF (CNF).
NFV Orchestrator (NFVO)	Manages the VNF lifecycle and Cloud Infrastructure resources (supported by the VIM) to ensure an optimised allocation of the necessary resources and connectivity.
Network Function Virtualisation (NFV)	The concept of separating network functions from the hardware they run on by using a virtual hardware abstraction layer.
Network Function Virtualisation Infrastructure (NFVI)	The totality of all hardware and software components used to build the environment in which a set of virtual applications (VAs) are deployed; also referred to as cloud infrastructure. The NFVI can span across many locations, e.g., places where data centres or edge nodes are operated. The network providing connectivity between these locations is regarded to be part of the cloud infrastructure. NFVI and VNF are the top-level conceptual entities in the scope of Network Function Virtualisation. All other components are sub-entities of these two main entities.
Network Service (NS)	Composition of Network Function(s) and/or Network Service(s), defined by its functional and behavioural specification, including the service lifecycle.
Open Network Automation Platform (ONAP)	An LFN project developing a comprehensive platform for orchestration, management, and automation of network and edge computing services for network operators, cloud providers, and enterprises.
ONAP OpenLab	ONAP community lab.
Open Platform for NFV (OPNFV)	A collaborative project under the Linux Foundation. OPNFV is now part of the LFN Anuket project. It aims to implement, test, and deploy tools for conformance and performance of NFV infrastructure.
OPNFV Verification Program (OVP)	An open-source, community-led compliance and verification program aiming to demonstrate the readiness and availability of commercial NFV products and services using OPNFV and ONAP components.
Platform	A cloud capabilities type in which the cloud service user can deploy, manage and run customer-created or customer-acquired applications using one or more programming languages and one or more execution environments supported by the cloud service provider. Adapted from ITU-T Y.3500. This includes the physical infrastructure, Operating Systems, virtualisation/containerisation software and other orchestration, security, monitoring/logging and life-cycle management software.
Pod	The smallest and simplest Kubernetes object. A Pod represents a set of running containers on a cluster. A Pod is typically set up to run a single primary container. It can also run optional sidecar containers that add supplementary features like logging.
Prometheus	An open-source monitoring and alerting system.
Quota	An imposed upper limit on specific types of resources, usually used to prevent excessive resource consumption by a given consumer (tenant, VM, container).
Resource pool	A logical grouping of cloud infrastructure hardware and software resources. A resource pool can be based on a certain resource type (for example, compute, storage and network) or a combination of resource types. A Cloud Infrastructure resource can be part of none, one or more resource pools.
Simultaneous Multithreading (SMT)	Simultaneous multithreading (SMT) is a technique for improving the overall efficiency of superscalar CPUs with hardware multithreading. SMT permits multiple independent threads of execution on a single core to better utilise the resources provided by modern processor architectures.
Tenant	Cloud service users sharing access to a set of physical and virtual resources, ITU-T Y.3500. Tenants represent an independently manageable logical pool of compute, storage and network resources abstracted from physical hardware.
Tenant Instance	Refers to an Instance owned by or dedicated for use by a single Tenant.
Tenant (Internal) Networks	Virtual networks that are internal to Tenant Instances.
User	Natural person, or entity acting on their behalf, associated with a cloud service customer that uses cloud services. Examples of such entities include devices and applications.
Virtual CPU (vCPU)	Represents a portion of the host’s computing resources allocated to a virtualised resource, for example, to a virtual machine or a container. One or more vCPUs can be assigned to a virtualised resource.
Virtualised Infrastructure Manager (VIM)	Responsible for controlling and managing the Network Function Virtualisation Infrastructure (NFVI) compute, storage and network resources.
Virtual Machine (VM)	Virtualised computation environment that behaves like a physical computer/server. A VM consists of all of the components (processor (CPU), memory, storage, interfaces/ports, etc.) of a physical computer/server. It is created using sizing information or Compute Flavour.
Virtualised Network Function (VNF)	A software implementation of a Network Function, capable of running on the Cloud Infrastructure. VNFs are built from one or more VNF Components (VNFC) and, in most cases, the VNFC is hosted on a single VM or Container.
Workload	An application (for example VNF, or CNF) that performs certain task(s) for the users. In the Cloud Infrastructure, these applications run on top of compute resources such as VMs or Containers.

Abbreviations¶

Term	Description
API	Application Programming Interface
BGP VPN	Border gateway Protocol Virtual Private network
CaaS	Containers as a Service
CI/CD	Continuous Integration/Continuous Deployment
CNF	Containerised Network Function
CNTT	Cloud iNfrastructure Telco Task Force
CPU	Central Processing Unit
DNS	Domain Name System
DPDK	Data Plane Development Kit
DHCP	Dynamic Host Configuration Protocol
ECMP	Equal Cost Multi-Path routing
ETSI	European Telecommunications Standards Institute
FPGA	Field Programmable Gate Array
MB/GB/TB	MegaByte/GigaByte/TeraByte
GPU	Graphics Processing Unit
GRE	Generic Routing Encapsulation
GSM	Global System for Mobile Communications (originally Groupe Spécial Mobile)
GSMA	GSM Association
GSLB	Global Service Load Balancer
GUI	Graphical User Interface
HA	High Availability
HDD	Hard Disk Drive
HTTP	Hypertext Transfer Protocol
HW	Hardware
IaaC (also IaC)	Infrastructure as a Code
IaaS	Infrastructure as a Service
ICMP	Internet Control Message Protocol
IMS	IP Multimedia Sub System
IO	Input/Output
IOPS	Input/Output per Second
IPMI	Intelligent Platform Management Interface
KVM	Kernel-based Virtual Machine
LCM	Life Cycle Management
LDAP	Lightweight Directory Access Protocol
LFN	Linux Foundation Networking
LMA	Logging, Monitoring and Analytics
LVM	Logical Volume Management
MANO	Management And Orchestration
MLAG	Multi-chassis Link Aggregation Group
NAT	Network Address Translation
NFS	Network File System
NFV	Network Function Virtualisation
NFVI	Network Function Virtualisation Infrastructure
NIC	Network Interface Card
NPU	Numeric Processing Unit
NTP	Network Time Protocol
NUMA	Non-Uniform Memory Access
OAI	Open Air Interface
OS	Operating System
OSTK	OpenStack
OPNFV	Open Platform for NFV
OVS	Open vSwitch
OWASP	Open Web Application Security Project
PCIe	Peripheral Component Interconnect Express
PCI-PT	PCIe Passthrough
PXE	Preboot Execution Environment
QoS	Quality of Service
RA	Reference Architecture
RA-2	Reference Architecture 2 (i.e., Reference Architecture for Kubernetes-based Cloud Infrastructure)
RBAC	Role-based Access Control
RBD	RADOS Block Device
REST	Representational state transfer
RI	Reference Implementation
RM	Reference Model
SAST	Static Application Security Testing
SDN	Software Defined Networking
SFC	Service Function Chaining
SG	Security Group
SLA	Service Level Agreement
SMP	Symmetric Multiprocessing
SMT	Simultaneous Multithreading
SNAT	Source Network Address Translation
SNMP	Simple Network Management Protocol
SR-IOV	Single Root Input Output Virtualisation
SSD	Solid State Drive
SSL	Secure Sockets Layer
SUT	System Under Test
TCP	Transmission Control Protocol
TLS	Transport Layer Security
ToR	Top of Rack
TPM	Trusted Platform Module
UDP	User Data Protocol
VIM	Virtualised Infrastructure Manager
VLAN	Virtual LAN
VM	Virtual Machine
VNF	Virtual Network Function
VRRP	Virtual Router Redundancy Protocol
VTEP	VXLAN Tunnel End Point
VXLAN	Virtual Extensible LAN
WAN	Wide Area Network
ZTA	Zero Trust Architecture

Conventions¶

The key words “MUST”, “MUST NOT”, “required”, “SHALL”, SHALL NOT”, “SHOULD”, “SHOULD NOT”, “recommended”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in RFC 2119 [246].

References¶

1: Network Functions Virtualisation (NFV); Terminology for Main Concepts in NFV. ETSI GR NFV 003 V1.5.1, January 2020. URL: https://www.etsi.org/deliver/etsi_gr/NFV/001_099/003/01.05.01_60/gr_NFV003v010501p.pdf.
2: Scott O. Bradner. Key words for use in RFCs to Indicate Requirement Levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
3: NGMN Overview on 5GRAN Functional Decomposition. 2018. URL: https://www.ngmn.org/wp-content/uploads/Publications/2018/180226_NGMN_RANFSX_D1_V20_Final.pdf.
4: Front haul Interoperability Test Specification (IOT). ORAN-WG4.IOT.0-v01.00. URL: https://static1.squarespace.com/static/5ad774cce74940d7115044b0/t/5db36ffa820b8d29022b6d08/1572040705841/ORAN-WG4.IOT.0-v01.00.pdf/2018/180226_NGMN_RANFSX_D1_V20_Final.pdf.
5: Network Functions Virtualisation (NFV) Release 3; Architecture; Report on the Enhancements of the NFV architecture towards "Cloud-native" and "PaaS". ETSI GR NFV-IFA 029 v3.3.1. URL: https://www.etsi.org/deliver/etsi_gr/NFV-IFA/001_099/029/03.03.01_60/gr_NFV-IFA029v030301p.pdf.
6: Jim Martin, Jack Burbank, William Kasch, and Professor David L. Mills. Network Time Protocol Version 4: Protocol and Algorithms Specification. RFC 5905, June 2010. URL: https://www.rfc-editor.org/info/rfc5905, doi:10.17487/RFC5905.
7: Professor David L. Mills and Brian Haberman. Network Time Protocol Version 4: Autokey Specification. RFC 5906, June 2010. URL: https://www.rfc-editor.org/info/rfc5906, doi:10.17487/RFC5906.
8: Chris Elliott, Brian Haberman, and Heiko Gerstung. Definitions of Managed Objects for Network Time Protocol Version 4 (NTPv4). RFC 5907, June 2010. URL: https://www.rfc-editor.org/info/rfc5907, doi:10.17487/RFC5907.
9: Benoit Lourdelet and Richard Gayraud. Network Time Protocol (NTP) Server Option for DHCPv6. RFC 5908, June 2010. URL: https://www.rfc-editor.org/info/rfc5908, doi:10.17487/RFC5908.
10: Timing characteristics of a synchronous equipment slave clock. ITU-T G.8262. URL: https://www.itu.int/rec/T-REC-G.8262.
11: Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network. ITU-T G.8275.2. URL: https://www.itu.int/rec/T-REC-G.8275.2.
12: O-RAN Acceleration Abstraction Layer General Aspects and Principles. O-RAN.WG6.AAL-GAnP-v01.00, November 2020. URL: https://www.o-ran.org.
13: WG6: Cloudification and Orchestration Workgroup specfications. URL: https://www.o-ran.org.
14: Energy efficiency measurement and metrics for telecommunication networks. ITU-T L.1330. URL: https://www.itu.int/rec/T-REC-L.1330.
15: Energy Efficiency measurement methodology and KPI/metrics for NFV. ETSI EN 303 471. URL: https://portal.etsi.org/webapp/workprogram/Report_WorkItem.asp?WKI_ID=50095.
16: Measurement method for energy efficiency of Network Functions Virtualisation (NFV) in laboratory environment. ETSI ES 203 539. URL: https://portal.etsi.org/webapp/workprogram/Report_WorkItem.asp?WKI_ID=47210.
17: Measurement method for energy efficiency of network functions virtualization. ITU-T L.1361. URL: https://www.itu.int/rec/T-REC-L.1361.
18: DMTF RedFish Specification. URL: https://www.dmtf.org/sites/default/files/standards/documents/DSP0268_2022.2.pdf.
19: Open RAN Technical Priority - Focus on Energy Efficiency. March 2022. URL: https://www.o-ran.org/ecosystem-resources.
20: QuEST Forum - NFV Workload Efficiency Whitepaper. October 2016. URL: https://tl9000.org/resources/documents/NFV%20Workload%20Efficiency%20Whitepaper.pdf.
21: Testing; NFVI Compute and Network Metrics Specification. ETSI GS NFV-TST 008 v3.2.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/008/03.02.01_60/gs_NFV-TST008v030201p.pdf.
22: Management and Orchestration; Performance Measurements Specification. ETSI GS NFV-IFA 027 v2.4.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/027/02.04.01_60/gs_nfv-ifa027v020401p.pdf.
23: Management and Orchestration; Report on Os-Ma-Nfvo reference point - application and service management use cases and recommendations. ETSI GR NFV IFA-012. URL: https://www.etsi.org/deliver/etsi_gr/NFV-IFA/001_099/012/03.01.01_60/gr_NFV-IFA012v030101p.pdf.
24: Operator Platform Telco Edge Requirements. GSMA OPG.02. URL: https://www.gsma.com/futurenetworks/operator-platform-hp/.
25: Acceleration Technologies; VNF Interfaces Specification. ETSI GS NFV-IFA 002 v2.1.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/002/02.01.01_60/gs_NFV-IFA002v020101p.pdf.
26: Network Functions Virtualisation (NFV); Acceleration Technologies; Acceleration Resource Management Interface Specification. ETSI NFV-IFA 019 V3.1.1. URL: https://www.etsi.org/deliver/etsi_gs/nfv-ifa/001_099/019/03.01.01_60/gs_nfv-ifa019v030101p.pdf.
27: Network Functions Virtualisation (NFV); Infrastructure; Hypervisor Domain. ETSI GS NFV-INF 004 V1.1.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-INF/001_099/004/01.01.01_60/gs_NFV-INF004v010101p.pdf.
28: Baseline Security Controls. GSMA FS.31 version 2.0. URL: https://www.gsma.com/security/resources/fs-31-gsma-baseline-security-controls/.
29: Network Equipment Security Assurance Scheme – Development and Lifecycle Security Requirements. GSMA FS.16 v2.2. URL: https://www.gsma.com/security/resources/fs-16-network-equipment-security-assurance-scheme-development-and-lifecycle-security-requirements/.
30: Open Networking & the Security of Open Source Software Deployment. URL: https://www.gsma.com/futurenetworks/resources/open-networking-the-security-of-open-source-software-deployment/.
31: 5G Security Guide. GSMA FS.40-v02.00, October 2021. URL: https://infocentre2.gsma.com/gp/wg/FSG/OfficialDocuments/FS.40%205G%20Security%20Guide%20v2.0%20(Current)/FS.40%20v2.0.pdf.
32: The Six Pillars of DevSecOps: Automation. 2020. URL: https://safecode.org/resource-secure-development-practices/the-six-pillars-of-devsecops-automation/.
33: Information technology — Security techniques — Information security management systems — Overview and vocabulary. ISO/IEC 27000:2018, 2018. URL: https://www.iso.org/standard/73906.html.
34: Information Security Management through Reflexive Security. 2019. URL: https://cloudsecurityalliance.org/artifacts/information-security-management-through-reflexive-security/.
35: Zero Trust Architecture (ZTA). NIST SP 800-207. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf.
36: Edge Computing: Next Steps in Architecture, Design and Testing. URL: https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing/.
37: CYBER; Trusted Cross-Domain Interface: Interface to offload sensitive functions to a trusted domain. ETSI TS 103 457, October 2018. URL: https://www.etsi.org/deliver/etsi_ts/103400_103499/103457/01.01.01_60/ts_103457v010101p.pdf.
38: Cloud infrastructure reference model. GSMA PRD NG.126 v3.0, 2022.
39: Openstack documentation. URL: https://docs.openstack.org/.
40: Network functions virtualisation (nfv); infrastructure overview. ETSI GS NFV-INF 001 V1.1.1, January 2015. URL: https://www.etsi.org/deliver/etsi_gs/NFV-INF/001_099/001/01.01.01_60/gs_NFV-INF001v010101p.pdf.
41: Openstack use cases. URL: https://docs.openstack.org/arch-design/use-cases.html.
42: Open vswitch. URL: https://www.openvswitch.org/.
43: OpenStack Community. Openstack wallaby projects. URL: https://docs.openstack.org/wallaby/projects.html.
44: Scott O. Bradner. Key words for use in rfcs to indicate requirement levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
45: H. Philip White. Center for internet security - password policy guide. 2020. URL: https://www.cisecurity.org/insights/white-papers/cis-password-policy-guide.
46: Center for internet security - controls v7.1. URL: https://www.cisecurity.org/controls/cis-controls-list.
47: Openstack - cpu dedicated set. URL: https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu_dedicated_set.
48: Openstack - cpu topologies. URL: https://docs.openstack.org/nova/latest/admin/cpu-topologies.html.
49: Openstack - neutron plugins and drivers. URL: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers.
50: Openstack - tags. URL: https://specs.openstack.org/openstack/api-wg/guidelines/tags.html.
51: Openstack - configuring the stateful services. URL: https://docs.openstack.org/ha-guide/control-plane-stateful.html.
52: Openstack - senlin documentation. URL: https://docs.openstack.org/senlin/wallaby/.
53: Openstack - neutron ovs agent support for baremetal with smart nic. URL: https://specs.openstack.org/openstack/neutron-specs/specs/stein/neutron-ovs-agent-support-baremetal-with-smart-nic.html.
54: National telecommunications and information administration - software bill of materials. URL: https://ntia.gov/SBOM.
55: Center for internet security. URL: https://www.cisecurity.org/.
56: Cloud security alliance. URL: https://cloudsecurityalliance.org/.
57: Open web application security project - cheat sheet series. URL: https://github.com/OWASP/CheatSheetSeries.
58: Open web application security project. URL: https://www.owasp.org.
59: Open web application security project - top ten security risks. URL: https://owasp.org/www-project-top-ten/.
60: Open web application security project - software maturity model (samm). URL: https://owaspsamm.org/blog/2019/12/20/version2-community-release/.
61: Open web application security project - web security testing guide. URL: https://github.com/OWASP/wstg/tree/master/document.
62: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 27001:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en.
63: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 27002:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27002:ed-2:v1:en.
64: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 7032:2012. 2012. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en.
65: Openstack storage. URL: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html#table-openstack-storage.
66: Openstack cinder driver support matrix. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.
67: Tungsten fabric - multicloud multistack sdn. URL: https://tungsten.io.
68: Openstack glossary. URL: https://docs.openstack.org/doc-contrib-guide/common/glossary.html.
69: Openstack feature support matrix. URL: https://docs.openstack.org/nova/latest/user/support-matrix.html.
70: Openstack storage architecture design. URL: https://docs.openstack.org/arch-design/design-storage.html.
71: Openstack nova: kvm. URL: https://docs.openstack.org/nova/wallaby/admin/configuration/hypervisor-kvm.html.
72: Openstack - hardening the virtualization layers. URL: https://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html.
73: Openstack reference architecture for 100, 300 and 500 nodes. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html.
74: (dpdk) release notes. URL: http://doc.dpdk.org/guides/rel_notes.
75: (dpdk) performance reports. URL: http://core.dpdk.org/perf-reports/.
76: Robert Moskowitz, Daniel Karrenberg, Yakov Rekhter, Eliot Lear, and Geert Jan de Groot. Address allocation for private internets. RFC 1918, February 1996. URL: https://www.rfc-editor.org/info/rfc1918, doi:10.17487/RFC1918.
77: Openstack - introducing octavia. URL: https://docs.openstack.org/octavia/latest/reference/introduction.html.
78: Openstack octavia (load-balancer service). URL: https://governance.openstack.org/tc/reference/projects/octavia.html.
79: Openstack/neutron-vpnaas. URL: https://opendev.org/openstack/neutron-vpnaas.
80: Openstack neutron: plugins. URL: https://wiki.openstack.org/wiki/Neutron#Plugins.
81: Openstack neutron: api extensions. URL: https://docs.openstack.org/neutron/latest/contributor/internals/api_extensions.html.
82: Openstack networking api v2.0: list extensions. URL: https://docs.openstack.org/api-ref/network/v2/#list-extensions.
83: Openstack networking api v2.0: show extension details. URL: https://docs.openstack.org/api-ref/network/v2/#show-extension-details.
84: Openstack neutron/ml2. URL: https://wiki.openstack.org/wiki/Neutron/ML2.
85: Openstack cinder driver support matrix. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.
86: Openstack (cinder) available drivers. URL: https://docs.openstack.org/cinder/latest/drivers.html.
87: Openstack cinder service configuration. URL: https://docs.openstack.org/cinder/latest/configuration/index.html.
88: Openstack cinder administration. URL: https://docs.openstack.org/cinder/latest/admin/index.html.
89: Ceph - the future of storage. URL: https://ceph.io/en.
90: Keystone, the openstack identity service. URL: https://docs.openstack.org/keystone/wallaby/.
91: Openstack - welcome to glance's documentation! URL: https://docs.openstack.org/glance/wallaby/.
92: Openstack block storage (cinder) documentation. URL: https://docs.openstack.org/cinder/wallaby/.
93: Openstack - welcome to swift's documentation! URL: https://docs.openstack.org/swift/wallaby/.
94: Openstack - welcome to neutron's documentation! URL: https://docs.openstack.org/neutron/wallaby/.
95: Openstack - scenario: high availability using distributed virtual routing (dvr). URL: https://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html.
96: Openstack neutron: distributed virtual routing with vrrp. URL: https://docs.openstack.org/neutron/wallaby/admin/config-dvr-ha-snat.html.
97: Openstack compute (nova). URL: https://docs.openstack.org/nova/wallaby/.
98: Openstack - welcome to ironic's documentation! URL: https://docs.openstack.org/ironic/wallaby/.
99: Openstack ironic api reference: bare metal api. URL: https://docs.openstack.org/api-ref/baremetal/.
100: Openstack - welcome to the heat documentation! URL: https://docs.openstack.org/heat/wallaby/.
101: Horizon: the openstack dashboard project. URL: https://docs.openstack.org/horizon/wallaby/.
102: Openstack - placement. URL: https://docs.openstack.org/placement/wallaby/index.html.
103: Openstack - placement: modeling with provider trees. URL: https://docs.openstack.org/placement/latest/user/provider-tree.html.
104: Openstack - placement usage. URL: https://docs.openstack.org/placement/latest/user/index.html.
105: Openstack key manager (barbican). URL: https://docs.openstack.org/barbican/wallaby/.
106: Openstack accelerator (cyborg). URL: https://docs.openstack.org/cyborg/wallaby/.
107: Openstack compute api guide 2.1.0: server concepts. URL: https://docs.openstack.org/api-guide/compute/server_concepts.html.
108: Openstack cyborg support matrix (wallaby). URL: https://docs.openstack.org/cyborg/wallaby/reference/support-matrix.html.
109: Openstack cyborg support matrix. URL: https://docs.openstack.org/cyborg/latest/reference/support-matrix.html.
110: Openstack cyborg architecture. URL: https://docs.openstack.org/cyborg/latest/user/architecture.html.
111: Openstack cyborg setup.cfg (wallaby). URL: https://opendev.org/openstack/cyborg/src/branch/stable/wallaby/setup.cfg.
112: Kubernetes. URL: https://kubernetes.io.
113: Reference architecture for kubernetes based cloud infrastructure. Anuket RA2. URL: https://cntt.readthedocs.io/projects/ra2/en/latest/index.html.
114: Openstack nova: flavors. URL: https://docs.openstack.org/nova/latest/user/flavors.html.
115: Open glossary of edge computing. URL: https://github.com/State-of-the-Edge/glossary/blob/master/edge-glossary.md.
116: Openstack - edge computing: next steps in architecture, design and testing. URL: https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing.
117: Openstack reference architecture for 100, 300 and 500 nodes: services placement summary. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html#services-placement-summary.
118: Openstack nova: image pre-caching. URL: https://docs.openstack.org/nova/latest/admin/image-caching.html#image-pre-caching.
119: Airship v2. URL: https://www.airshipit.org/.
120: Starlingx - deploy your edge cloud now. URL: https://www.starlingx.io/.
121: Openstack tripleo. URL: http://opendev.org/openstack/tripleo-common.
122: Openstack compute microversions. URL: https://docs.openstack.org/api-guide/compute/microversions.html.
123: Identity api v3. URL: https://docs.openstack.org/api-ref/identity/v3/index.html.
124: Identity api v3 extensions. URL: https://docs.openstack.org/api-ref/identity/v3-ext/.
125: Security compliance and pci-dss. URL: https://docs.openstack.org/keystone/wallaby/admin/configuration.html#security-compliance-and-pci-dss.
126: Image service api. URL: https://docs.openstack.org/api-ref/image/v2/.
127: Image service versions. URL: https://docs.openstack.org/api-ref/image/versions/index.html#version-history.
128: Block storage api. URL: https://docs.openstack.org/api-ref/block-storage/.
129: Cinder rest api version history. URL: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html.
130: Object storage api. URL: https://docs.openstack.org/api-ref/object-store/index.html.
131: Discoverability. URL: https://docs.openstack.org/swift/latest/api/discoverability.html.
132: Networking service apis. URL: https://docs.openstack.org/api-ref/network/.
133: Networking api v2.0. URL: https://docs.openstack.org/api-ref/network/v2/.
134: Compute api. URL: https://docs.openstack.org/api-ref/compute/.
135: Compute rest api version history. URL: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html.
136: Placement api. URL: https://docs.openstack.org/api-ref/placement/.
137: Placement rest api version history. URL: https://docs.openstack.org/placement/latest/placement-api-microversion-history.html.
138: Orchestration service api. URL: https://docs.openstack.org/api-ref/orchestration/.
139: Template version history. URL: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html.
140: Heat orchestration template (hot) specification. URL: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky.
141: Openstack apis. URL: https://docs.openstack.org/api-ref/.
142: Kubernetes apis. URL: https://kubernetes.io/docs/concepts/overview/kubernetes-api/.
143: Kvm apis. URL: https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt.
144: Libvirt apis. URL: https://libvirt.org/html/index.html.
145: Barbican api. URL: https://docs.openstack.org/barbican/latest/api/.
146: Openstack - security boundaries and threats. URL: https://docs.openstack.org/security-guide/introduction/security-boundaries-and-threats.html.
147: Openstack security guide. URL: https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html.
148: Mitre - common vulnerabilities and exposures. URL: https://cve.mitre.org/.
149: National institute of standards and technology vulnerabilities metrics. URL: https://nvd.nist.gov/vuln-metrics/cvss.
150: Openstack security guide - identity. URL: https://docs.openstack.org/security-guide/identity.html.
151: Openstack security guide - authentication methods. URL: https://docs.openstack.org/security-guide/identity/authentication-methods.html.
152: Openstack security guide - policies. URL: https://docs.openstack.org/security-guide/identity/policies.html#policy-section.
153: Openstack keystone default roles. URL: https://docs.openstack.org/keystone/latest/admin/service-api-protection.html.
154: Openstack - introduction to tls and ssl. URL: https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html.
155: Center for internet security cis-cat pro. URL: https://www.cisecurity.org/cybersecurity-tools/cis-cat-pro/.
156: Center for internet security benchmarks. URL: https://www.cisecurity.org/cis-benchmarks/.
157: Openstack image signature verification. URL: https://docs.openstack.org/glance/wallaby/user/signature.html.
158: Openstack - sr-iov passthrough for networking. URL: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking.
159: Openstack trusted images. URL: https://docs.openstack.org/security-guide/instance-management/security-services-for-instances.html#trusted-images.
160: Openstack virtual machine image guide. URL: https://docs.openstack.org/image-guide/.
161: Adding signed images. URL: https://docs.openstack.org/operations-guide/ops-user-facing-operations.html#adding-signed-images.
162: Network functions virtualisation (nfv) release 4; protocols and data models; vnf package and pnfd archive specification. ETSI GS NFV-SOL 004 V4.3.1, July 2022. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/004/04.03.01_60/gs_NFV-SOL004v040301p.pdf.
163: Network functions virtualisation (nfv) release 2; security; vnf package security specification. ETSI GS NFV-SEC 021 V2.6.1, June 2019. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/021/02.06.01_60/gs_nfv-sec021v020601p.pdf.
164: Foreman. URL: https://www.theforeman.org/.
165: Ansible documentation. URL: https://docs.ansible.com/.
166: Openstack tripleo architecture. URL: https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html#project-architecture.
167: Ovp. URL: https://www.opnfv.org/verification.
168: Jenkins. URL: https://build.opnfv.org/.
169: Test dabase. URL: https://docs.opnfv.org/en/stable-hunter/_images/OPNFV_testing_working_group.png.
170: S3 compatible storage service. URL: http://artifacts.opnfv.org/.
171: Functest-wallaby-zip. URL: https://build.opnfv.org/ci/job/functest-wallaby-zip/4/console.
172: Xtesting ci. URL: https://galaxy.ansible.com/collivier/xtesting.
173: Docker. URL: https://www.docker.com/.
174: Xtesting. URL: https://xtesting.readthedocs.io/en/latest/.
175: Opnfv fraser. URL: https://www.sdxcentral.com/articles/news/opnfvs-6th-release-brings-testing-capabilities-that-orange-is-already-using/2018/05/.
176: Xtesting python package. URL: https://pypi.org/project/xtesting/.
177: Test case execution description. URL: https://git.opnfv.org/functest-xtesting/tree/docker/core/testcases.yaml.
178: Ci/cd toolchains in a few commands. URL: https://github.com/collivier/ansible-role-xtesting#readme.
179: Ci/cd deployment models. URL: https://lists.opnfv.org/g/opnfv-tsc/message/5702.
180: Anuket releng. URL: https://git.opnfv.org/releng/tree/jjb/functest.
181: Test case result dump. URL: http://artifacts.opnfv.org/functest/9ID39XK47PMZ.zip.
182: Xtesting samples. URL: https://git.opnfv.org/functest-xtesting/plain/ansible/site.yml?h=stable/wallaby.
183: Openstack verification. URL: https://git.opnfv.org/functest/plain/ansible/site.yml?h=stable/wallaby.
184: Anuket rc1. URL: https://git.opnfv.org/functest/plain/ansible/site.cntt.yml?h=stable/wallaby.
185: Kubernetes verification. URL: https://git.opnfv.org/functest-kubernetes/plain/ansible/site.yml?h=stable/v1.22.
186: Functest. URL: https://functest.readthedocs.io/en/stable-wallaby/.
187: Refstack. URL: https://refstack.openstack.org/.
188: Networking bgpvpn. URL: https://docs.openstack.org/networking-bgpvpn/latest/.
189: Networking sfc. URL: https://docs.openstack.org/networking-sfc/latest/.
190: Devstack gates. URL: https://docs.opendev.org/opendev/system-config/latest/devstack-gate.html.
191: Rally. URL: https://github.com/openstack/rally-openstack.
192: Temptest. URL: https://github.com/openstack/tempest.
193: Temptest. URL: https://docs.openstack.org/devstack/latest/.
194: Raspberry pi. URL: https://www.raspberrypi.org/.
195: Functest daily jobs. URL: https://build.opnfv.org/ci/view/functest/job/functest-wallaby-daily/17/.
196: Openstack performance tools. URL: https://docs.openstack.org/developer/performance-docs/methodologies/tools.html.
197: Run alpine functest containers (wallaby). URL: https://wiki.anuket.io/display/HOME/Functest+Wallaby.
198: Functest gates. URL: https://build.opnfv.org/ci/view/functest.
199: New functest cntt containers. URL: https://lists.opnfv.org/g/opnfv-tsc/message/5717.
200: Keystone-tempest-plugin. URL: https://opendev.org/openstack/keystone-tempest-plugin.
201: Functest smoke cntt. URL: https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable%2Fwallaby.
202: Cinder-tempest-plugin. URL: https://opendev.org/openstack/cinder-tempest-plugin.
203: Functest review 68881. URL: https://gerrit.opnfv.org/gerrit/68881.
204: Functest review 71011. URL: https://gerrit.opnfv.org/gerrit/71011.
205: Openstack bug 1770179. URL: https://launchpad.net/bugs/1770179.
206: Openstack bug 1677525. URL: https://launchpad.net/bugs/1677525.
207: Openstack bug 1317133. URL: https://launchpad.net/bugs/1317133.
208: Openstack bug 1905432. URL: https://launchpad.net/bugs/1905432.
209: Neutron-tempest-plugin. URL: https://opendev.org/openstack/neutron-tempest-plugin.
210: Openstack bug 1863707. URL: https://launchpad.net/bugs/1863707.
211: Functest review 69105. URL: https://gerrit.opnfv.org/gerrit/69105.
212: Openstack bug 1676207. URL: https://launchpad.net/bugs/1676207.
213: Openstack bug 1836595. URL: https://launchpad.net/bugs/1836595.
214: Openstack bug 1186354. URL: https://launchpad.net/bugs/1186354.
215: Openstack bug 1014647. URL: https://launchpad.net/bugs/1014647.
216: Openstack bug 1311500. URL: https://launchpad.net/bugs/1311500.
217: Openstack bug 1161411. URL: https://launchpad.net/bugs/1161411.
218: Openstack bug 1540645. URL: https://launchpad.net/bugs/1540645.
219: Heat-tempest-plugin. URL: https://opendev.org/openstack/heat-tempest-plugin.
220: Openstack story 2007804. URL: https://storyboard.openstack.org/#!/story/2007804.
221: Functest review 69926. URL: https://gerrit.opnfv.org/gerrit/69926.
222: Functest review 69931. URL: https://gerrit.opnfv.org/gerrit/69931.
223: Functest review 70004. URL: https://gerrit.opnfv.org/gerrit/70004.
224: Tempest-horizon. URL: https://github.com/openstack/tempest-horizon.
225: Functest healthcheck. URL: https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable%2Fwallaby.
226: Functest benchmarking cntt. URL: https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable%2Fwallaby.
227: Rally_full_cntt. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally_full_cntt-run-5/rally_full_cntt/rally_full_cntt.html.
228: Rally_jobs_cntt. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally_jobs_cntt-run-5/rally_jobs_cntt/rally_jobs_cntt.html.
229: Vmtp. URL: http://vmtp.readthedocs.io/en/latest.
230: Shaker. URL: https://pyshaker.readthedocs.io/en/latest/.
231: Vmtp scenarios. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.json.
232: Functest vmtp. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.html.
233: Shaker scenarios. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-shaker-run-8/shaker/report.json.
234: Functest vnf. URL: https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable%2Fwallaby.
235: Clearwater ims. URL: https://clearwater.readthedocs.io/en/stable/.
236: Vyos vrouter. URL: https://www.vyos.io/.
237: Openairinterface vepc. URL: https://www.openairinterface.org/.
238: Cloudify. URL: https://cloudify.co.
239: Juju. URL: https://jaas.ai/.
240: Clearwater-live-test. URL: https://github.com/Metaswitch/clearwater-live-test.
241: Docker http/https proxy. URL: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy.
242: Openstack autoscaling with heat. URL: https://docs.openstack.org/senlin/latest/scenarios/autoscaling_heat.html.
243: Reference model for cloud infrastructure (rm). GSMA PRD NG.126 v3.0, 2022.
244: Kubernetes documentation. URL: https://kubernetes.io/docs/home/.
245: What is kubernetes. URL: https://kubernetes.io/docs/concepts/overview/.
246: Scott O. Bradner. Key words for use in rfcs to indicate requirement levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
247: Cis password policy guide. URL: https://www.cisecurity.org/insights/white-papers/cis-password-policy-guide.
248: Cis controls list. URL: https://www.cisecurity.org/controls/cis-controls-list.
249: Cve - common vulnerabilities and exposures. URL: https://cve.mitre.org/.
250: Sbom - software bill of materials. URL: https://ntia.gov/page/software-bill-materials.
251: Cloud security alliance. URL: https://cloudsecurityalliance.org/.
252: Owasp cheat sheet series (ocss). URL: https://github.com/OWASP/CheatSheetSeries/.
253: Owasp top ten security risks. URL: https://owasp.org/www-project-top-ten/.
254: Owasp software maturity model (samm). URL: https://owaspsamm.org/.
255: Owasp web security testing guide. URL: https://github.com/OWASP/wstg/.
256: Iso/iec 27001. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en.
257: Iso/iec 27032. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en.
258: Cncf kubernetes conformance. URL: https://github.com/cncf/k8s-conformance.
259: Open container initiative (oci) runtime spec. URL: https://github.com/opencontainers/runtime-spec.
260: Kubernetes docs - pod. URL: https://kubernetes.io/docs/concepts/workloads/pods.
261: Kubernetes docs - replicaset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset.
262: Kubernetes docs - deployment. URL: https://kubernetes.io/docs/concepts/workloads/controllers/deployment.
263: Kubernetes docs - daemonset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset.
264: Kubernetes docs - job. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job.
265: Kubernetes docs - cronjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs.
266: Kubernetes docs - statefulset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset.
267: Kubernetes docs - cpu manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies.
268: Kubernetes docs - huge pages. URL: https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages.
269: Kubernetes docs - memory manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager.
270: Kubernetes docs - topology manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager.
271: Kubernetes node feature discovery. URL: https://kubernetes-sigs.github.io/node-feature-discovery/.
272: Kubernetes docs - device plugin framework. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.
273: Kubernetes sr-iov network device plugin. URL: https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin.
274: Github: multus-cni. URL: https://github.com/k8snetworkplumbingwg/multus-cni.
275: Kubernetes docs - network plugins. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.
276: Kubernetes network custom resource definition. URL: https://github.com/k8snetworkplumbingwg/multi-net-spec.
277: Tungsten fabric. URL: https://tungsten.io/.
278: Af_xdp device plugin. URL: https://github.com/intel/afxdp-plugins-for-kubernetes.
279: Cloud native data plane. URL: https://cndp.io/.
280: Dpdk af_xdp poll mode driver. URL: https://doc.dpdk.org/guides/nics/af_xdp.html.
281: Kubernetes docs - ingress. URL: https://kubernetes.io/docs/concepts/services-networking/ingress.
282: Kubernetes docs - service. URL: https://kubernetes.io/docs/concepts/services-networking/service.
283: Kubernetes docs - endpointslices. URL: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices.
284: Kubernetes docs - network policies. URL: https://kubernetes.io/docs/concepts/services-networking/network-policies.
285: Container network interface. URL: https://github.com/containernetworking/cni.
286: Kubernetes network special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-network.
287: Kubernetes network plumbing working group. URL: https://github.com/k8snetworkplumbingwg/community.
288: Helm documentation. URL: https://helm.sh/docs.
289: Kubernetes docs - custom resources. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources.
290: Kubernetes docs - custom resource definitions. URL: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.
291: Kubernetes docs - api server aggregation. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.
292: Kubernetes docs - custom controllers. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-controllers.
293: Kubernetes docs - operator pattern. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/operator.
294: Operator hub. URL: https://operatorhub.io.
295: Reference implementation based on ra2 specifications (ri2). URL: https://cntt.readthedocs.io/projects/ri2/.
296: Reference architecture (ra1) for openstack based cloud infrastructure. URL: https://cntt.readthedocs.io/projects/ra1/.
297: Node feature discovery. URL: https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html.
298: Kubernetes distributions and platforms document. URL: https://docs.google.com/spreadsheets/d/1uF9BoDzzisHSQemXHIKegMhuythuq_GL3N1mlUUK2h0/.
299: Kubernetes supported versions. URL: https://kubernetes.io/releases/version-skew-policy/#supported-versions.
300: Kubernetes alpha api. URL: https://kubernetes.io/docs/reference/using-api/#api-versioning.
301: Kubernetes publishing services (servicetypes). URL: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types.
302: Kubernetes ingress. URL: https://kubernetes.io/docs/concepts/services-networking/ingress/.
303: Open container initiative runtime specification, version 1.0.0. URL: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/spec.md.
304: Kubernetes: introducing container runtime interface (cri) in kubernetes. URL: https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/.
305: Kubernetes network custom resource definition de-facto standard. URL: https://github.com/k8snetworkplumbingwg/multi-net-spec/tree/master/v1.2.
306: Kubernetes csi drivers. URL: https://kubernetes-csi.github.io/docs/drivers.html.
307: Application service descriptor (asd) and packaging proposals for cnf. URL: https://wiki.onap.org/display/DW/Application+Service+Descriptor+%28ASD%29+and+packaging+Proposals+for+CNF.
308: Etsi gs nfv-sol 001: nfv descriptors based on tosca specification. ETSI GS NFV-SOL 001, 2022. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/001/04.02.01_60/gs_NFV-SOL001v040201p.pdf.
309: Kubernetes types of volumes, hostpath. URL: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath.
310: Cnf testsuite, rationale, test if the helm chart is published: helm_chart_published. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-helm-chart-is-published-helm_chart_published.
311: Cnf testsuite, rationale, test if the helm chart is valid: helm_chart_valid. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-helm-chart-is-valid-helm_chart_valid.
312: Cnf testsuite, rationale, to test if the cnf can perform a rolling update: rolling_update. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-the-cnf-can-perform-a-rolling-update-rolling_update.
313: Cnf testsuite, rationale, to check if a cnf version can be downgraded through a rolling_downgrade: rolling_downgrade. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-a-cnf-version-can-be-downgraded-through-a-rolling_downgrade-rolling_downgrade.
314: Cnf testsuite, rationale, to check if the cnf is compatible with different cnis: cni_compatibility. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-is-compatible-with-different-cnis-cni_compatibility.
315: Cnf testsuite, rationale, test if the cnf crashes when node drain occurs: node_drain. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-node-drain-occurs-node_drain.
316: Cnf testsuite, rationale, test if the cnf crashes when network latency occurs: pod_network_latency. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-network-latency-occurs-pod_network_latency.
317: Cnf testsuite, rationale, test if the cnf crashes when disk fill occurs: disk_fill. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-disk-fill-occurs-disk_fill.
318: Cnf testsuite, rationale, test if the cnf crashes when pod memory hog occurs: pod_memory_hog. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-memory-hog-occurs-pod_memory_hog.
319: Cnf testsuite, rationale, test if the cnf crashes when pod io stress occurs: pod_io_stress. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-io-stress-occurs-pod_io_stress.
320: Cnf testsuite, rationale, test if the cnf crashes when pod network corruption occurs: pod_network_corruption. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-corruption-occurs-pod_network_corruption.
321: Cnf testsuite, rationale, test if the cnf crashes when pod network duplication occurs: pod_network_duplication. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-duplication-occurs-pod_network_duplication.
322: Cnf testsuite, rationale, to test if the cnf uses local storage: no_local_volume_configuration. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-the-cnf-uses-local-storage-no_local_volume_configuration.
323: Cnf testsuite, rationale, to test if there is a liveness entry in the helm chart: liveness. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-is-a-liveness-entry-in-the-helm-chart-liveness.
324: Cnf testsuite, rationale, to test if there is a readiness entry in the helm chart: readiness. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-is-a-readiness-entry-in-the-helm-chart-readiness.
325: Cnf testsuite, rationale, to check if there is automatic mapping of service accounts: service_account_mapping. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-is-automatic-mapping-of-service-accounts-service_account_mapping.
326: Cnf testsuite, rationale, to check if there is a host network attached to a pod: host_network. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-is-a-host-network-attached-to-a-pod-host_network.
327: Cnf testsuite, rationale, to check if containers are running with hostpid or hostipc privileges: host_pid_ipc_privileges. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-are-running-with-hostpid-or-hostipc-privileges-host_pid_ipc_privileges.
328: Cnf testsuite, rationale, to check if containers have resource limits defined: resource_policies. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-have-resource-limits-defined-resource_policies.
329: Cnf testsuite, rationale, to check if containers have immutable file systems: immutable_file_systems. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-have-immutable-file-systems-immutable_file_systems.
330: Kubernetes documentation: images. URL: https://kubernetes.io/docs/concepts/containers/images/.
331: Cnf testsuite, rationale, to test if there are any (non-declarative) hardcoded ip addresses or subnet masks in the k8s runtime configuration: hardcoded_ip_addresses_in_k8s_runtime_configuration. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-are-any-non-declarative-hardcoded-ip-addresses-or-subnet-masks-in-the-k8s-runtime-configuration-hardcoded_ip_addresses_in_k8s_runtime_configuration.
332: Kubernetes documentation: service. URL: https://kubernetes.io/docs/concepts/services-networking/service/.
333: Kubernetes documentation: immutable configmaps. URL: https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable.
334: Kubernetes documentation: horizontal pod autoscaling. URL: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/.
335: Cnf testsuite, rationale, to check if the cnf has a reasonable image size: reasonable_image_size. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-has-a-reasonable-image-size-reasonable_image_size.
336: Cnf testsuite, rationale, to check if the cnf have a reasonable startup time: reasonable_startup_time. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-have-a-reasonable-startup-time-reasonable_startup_time.
337: Cnf testsuite, rationale, to check if there are any privileged containers: privileged_containers. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-are-any-privileged-containers-privileged_containers.
338: Cnf testsuite, rationale, to check if any containers are running as a root user (checks the user outside the container that is running dockerd): non_root_user. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-any-containers-are-running-as-a-root-user-checks-the-user-outside-the-container-that-is-running-dockerd-non_root_user.
339: Cnf testsuite, rationale, to check if any containers allow for privilege escalation: privilege_escalation. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-any-containers-allow-for-privilege-escalation-privilege_escalation.
340: Cnf testsuite, rationale, to check if containers are running with non-root user with non-root membership: non_root_containers. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-are-running-with-non-root-user-with-non-root-membership-non_root_containers.
341: Kubernetes documentation: recommended labels. URL: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/.
342: The twelve factor app: logs. URL: https://12factor.net/logs.
343: Cnf testsuite, rationale, to test if there are host ports used in the service configuration: hostport_not_used. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-are-host-ports-used-in-the-service-configuration-hostport_not_used.
344: Kubernetes documentation ports and protocols. URL: https://kubernetes.io/docs/reference/networking/ports-and-protocols/.
345: Building secure microservices-based applications using service-mesh architecture. NIST Special Publication 800-204A, 2020. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-204a.pdf.
346: Attribute-based access control for microservices-based applications using a service mesh. NIST Special Publication 800-204B, 2020. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-204B.pdf.
347: Arstechnica: tesla cloud resources are hacked to run cryptocurrency-mining malware. URL: https://arstechnica.com/information-technology/2018/02/tesla-cloud-resources-are-hacked-to-run-cryptocurrency-mining-malware/.
348: Kubernetes testing special interest group. URL: https://github.com/kubernetes/community/blob/master/sig-testing/charter.md.
349: Kubernetes end-to-end testing. URL: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md.
350: Kubernetes feature gates. URL: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates.
351: Kubernetes kep-3136. URL: https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/3136-beta-apis-off-by-default/README.md.
352: Kubernetes api. URL: https://kubernetes.io/docs/reference/using-api/.
353: Kubernetes api reference. URL: https://kubernetes.io/docs/reference/kubernetes-api/.
354: Kubernetes api groups. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/ #-strong-api-groups-strong-.
355: Kubernetes api machinery special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-api-machinery.
356: Kubernetes feature crossnamespacepodaffinity. URL: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#namespace-selector.
357: Kubernetes feature storageversionapi. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#storageversion-v1alpha1-internal-apiserver-k8s-io.
358: Kubernetes apps special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-apps.
359: Kubernetes feature daemonsetupdatesurge. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#rollingupdatedaemonset-v1-apps.
360: Kubernetes feature indexedjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job/.
361: Kubernetes feature statefulset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/.
362: Kubernetes feature suspendjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job/#suspending-a-job.
363: Kubernetes feature tainteviction. URL: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions.
364: Kubernetes feature ttlafterfinished. URL: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/.
365: Kubernetes auth special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-auth.
366: Kubernetes feature boundserviceaccounttokenvolume. URL: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md.
367: Kubernetes cluster lifecycle special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle.
368: Kubernetes instrumentation special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-instrumentation.
369: Kubernetes network special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-network.
370: Kubernetes feature ipv6dualstack. URL: https://kubernetes.io/docs/concepts/services-networking/dual-stack/.
371: Kubernetes node special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-node.
372: Kubernetes feature probeterminationgraceperiod. URL: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#probe-level-terminationgraceperiodseconds.
373: Kubernetes feature downwardapihugepages. URL: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.
374: Kubernetes feature podreadinessgate. URL: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate.
375: Kubernetes feature sysctls. URL: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/.
376: Kubernetes scheduling special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-scheduling.
377: Kubernetes feature localstoragecapacityisolation. URL: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.
378: Kubernetes storage special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-storage.
379: Alibaba cloud blog: what can we learn from twitter's move to kubernetes. URL: https://www.alibabacloud.com/blog/what-can-we-learn-from-twitters-move-to-kubernetes_595156.
380: Youtube: kubernetes failure stories, or: how to crash your cluster - henning jacobs. URL: https://www.youtube.com/watch?v=LpFApeaGv7A.
381: Cncf blog: demystifying kubernetes as a service – how alibaba cloud manages 10,000s of kubernetes clusters. URL: https://www.cncf.io/blog/2019/12/12/demystifying-kubernetes-as-a-service-how-does-alibaba-cloud-manage-10000s-of-kubernetes-clusters/.
382: Google docs: kep: multinetwork podnetwork object. URL: https://docs.google.com/document/d/17LhyXsEgjNQ0NWtvqvtgJwVqdJWreizsgAZHWflgP-A/edit.
383: Kubernetes docs: user namespaces. URL: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/.
384: Kep-127: support user namespaces in stateless pods. URL: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/127-user-namespaces.
385: Wikipedia: linux namespaces. URL: https://en.wikipedia.org/wiki/Linux_namespaces.