Gaps, Innovation, and Development¶

The purpose of this chapter is to identify the gaps between what is required for automated deployment of VNFs on Cloud Infrastructure frameworks and the framework offered by OpenStack. Once gaps are identified, the next step will be to propose a plan to address these gaps. The most obvious way to address the gaps will be to propose a set of APIs in the upstream OpenStack community

The Gap¶

Autoscaling¶

With regards to resource autoscaling (gen.scl.01 General Recommendations) it is recommended that the NFVO/VNFM manages the policy and triggers a scale-up or scale-down action based on application telemetry, event, AI, or ML etc. While the use of telemetry and alarming system can trigger a scaling operation based on resource utilisation, without application context this may not provide the granularity or reaction time required by the application. It is therefore suggested that an OpenStack scaling operation is called using an appropriate autoscaling web-hook by the NFVO/VNFM.

For more information on auto-scaling with Heat please see the OpenStack document “Autoscaling with heat [242]”. Please note that the OpenStack Senlin service is still under development with major architectural changes made in the OpenStack Ussuri release.

Please note: physical compute node autoscaling is out of scope.

1: Network Functions Virtualisation (NFV); Terminology for Main Concepts in NFV. ETSI GR NFV 003 V1.5.1, January 2020. URL: https://www.etsi.org/deliver/etsi_gr/NFV/001_099/003/01.05.01_60/gr_NFV003v010501p.pdf.
2: Scott O. Bradner. Key words for use in RFCs to Indicate Requirement Levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
3: NGMN Overview on 5GRAN Functional Decomposition. 2018. URL: https://www.ngmn.org/wp-content/uploads/Publications/2018/180226_NGMN_RANFSX_D1_V20_Final.pdf.
4: Front haul Interoperability Test Specification (IOT). ORAN-WG4.IOT.0-v01.00. URL: https://static1.squarespace.com/static/5ad774cce74940d7115044b0/t/5db36ffa820b8d29022b6d08/1572040705841/ORAN-WG4.IOT.0-v01.00.pdf/2018/180226_NGMN_RANFSX_D1_V20_Final.pdf.
5: Network Functions Virtualisation (NFV) Release 3; Architecture; Report on the Enhancements of the NFV architecture towards "Cloud-native" and "PaaS". ETSI GR NFV-IFA 029 v3.3.1. URL: https://www.etsi.org/deliver/etsi_gr/NFV-IFA/001_099/029/03.03.01_60/gr_NFV-IFA029v030301p.pdf.
6: Jim Martin, Jack Burbank, William Kasch, and Professor David L. Mills. Network Time Protocol Version 4: Protocol and Algorithms Specification. RFC 5905, June 2010. URL: https://www.rfc-editor.org/info/rfc5905, doi:10.17487/RFC5905.
7: Professor David L. Mills and Brian Haberman. Network Time Protocol Version 4: Autokey Specification. RFC 5906, June 2010. URL: https://www.rfc-editor.org/info/rfc5906, doi:10.17487/RFC5906.
8: Chris Elliott, Brian Haberman, and Heiko Gerstung. Definitions of Managed Objects for Network Time Protocol Version 4 (NTPv4). RFC 5907, June 2010. URL: https://www.rfc-editor.org/info/rfc5907, doi:10.17487/RFC5907.
9: Benoit Lourdelet and Richard Gayraud. Network Time Protocol (NTP) Server Option for DHCPv6. RFC 5908, June 2010. URL: https://www.rfc-editor.org/info/rfc5908, doi:10.17487/RFC5908.
10: Timing characteristics of a synchronous equipment slave clock. ITU-T G.8262. URL: https://www.itu.int/rec/T-REC-G.8262.
11: Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network. ITU-T G.8275.2. URL: https://www.itu.int/rec/T-REC-G.8275.2.
12: O-RAN Acceleration Abstraction Layer General Aspects and Principles. O-RAN.WG6.AAL-GAnP-v01.00, November 2020. URL: https://www.o-ran.org.
13: WG6: Cloudification and Orchestration Workgroup specfications. URL: https://www.o-ran.org.
14: Energy efficiency measurement and metrics for telecommunication networks. ITU-T L.1330. URL: https://www.itu.int/rec/T-REC-L.1330.
15: Energy Efficiency measurement methodology and KPI/metrics for NFV. ETSI EN 303 471. URL: https://portal.etsi.org/webapp/workprogram/Report_WorkItem.asp?WKI_ID=50095.
16: Measurement method for energy efficiency of Network Functions Virtualisation (NFV) in laboratory environment. ETSI ES 203 539. URL: https://portal.etsi.org/webapp/workprogram/Report_WorkItem.asp?WKI_ID=47210.
17: Measurement method for energy efficiency of network functions virtualization. ITU-T L.1361. URL: https://www.itu.int/rec/T-REC-L.1361.
18: DMTF RedFish Specification. URL: https://www.dmtf.org/sites/default/files/standards/documents/DSP0268_2022.2.pdf.
19: Open RAN Technical Priority - Focus on Energy Efficiency. March 2022. URL: https://www.o-ran.org/ecosystem-resources.
20: QuEST Forum - NFV Workload Efficiency Whitepaper. October 2016. URL: https://tl9000.org/resources/documents/NFV%20Workload%20Efficiency%20Whitepaper.pdf.
21: Testing; NFVI Compute and Network Metrics Specification. ETSI GS NFV-TST 008 v3.2.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/008/03.02.01_60/gs_NFV-TST008v030201p.pdf.
22: Management and Orchestration; Performance Measurements Specification. ETSI GS NFV-IFA 027 v2.4.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/027/02.04.01_60/gs_nfv-ifa027v020401p.pdf.
23: Management and Orchestration; Report on Os-Ma-Nfvo reference point - application and service management use cases and recommendations. ETSI GR NFV IFA-012. URL: https://www.etsi.org/deliver/etsi_gr/NFV-IFA/001_099/012/03.01.01_60/gr_NFV-IFA012v030101p.pdf.
24: Operator Platform Telco Edge Requirements. GSMA OPG.02. URL: https://www.gsma.com/futurenetworks/operator-platform-hp/.
25: Acceleration Technologies; VNF Interfaces Specification. ETSI GS NFV-IFA 002 v2.1.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-IFA/001_099/002/02.01.01_60/gs_NFV-IFA002v020101p.pdf.
26: Network Functions Virtualisation (NFV); Acceleration Technologies; Acceleration Resource Management Interface Specification. ETSI NFV-IFA 019 V3.1.1. URL: https://www.etsi.org/deliver/etsi_gs/nfv-ifa/001_099/019/03.01.01_60/gs_nfv-ifa019v030101p.pdf.
27: Network Functions Virtualisation (NFV); Infrastructure; Hypervisor Domain. ETSI GS NFV-INF 004 V1.1.1. URL: https://www.etsi.org/deliver/etsi_gs/NFV-INF/001_099/004/01.01.01_60/gs_NFV-INF004v010101p.pdf.
28: Baseline Security Controls. GSMA FS.31 version 2.0. URL: https://www.gsma.com/security/resources/fs-31-gsma-baseline-security-controls/.
29: Network Equipment Security Assurance Scheme – Development and Lifecycle Security Requirements. GSMA FS.16 v2.2. URL: https://www.gsma.com/security/resources/fs-16-network-equipment-security-assurance-scheme-development-and-lifecycle-security-requirements/.
30: Open Networking & the Security of Open Source Software Deployment. URL: https://www.gsma.com/futurenetworks/resources/open-networking-the-security-of-open-source-software-deployment/.
31: 5G Security Guide. GSMA FS.40-v02.00, October 2021. URL: https://infocentre2.gsma.com/gp/wg/FSG/OfficialDocuments/FS.40%205G%20Security%20Guide%20v2.0%20(Current)/FS.40%20v2.0.pdf.
32: The Six Pillars of DevSecOps: Automation. 2020. URL: https://safecode.org/resource-secure-development-practices/the-six-pillars-of-devsecops-automation/.
33: Information technology — Security techniques — Information security management systems — Overview and vocabulary. ISO/IEC 27000:2018, 2018. URL: https://www.iso.org/standard/73906.html.
34: Information Security Management through Reflexive Security. 2019. URL: https://cloudsecurityalliance.org/artifacts/information-security-management-through-reflexive-security/.
35: Zero Trust Architecture (ZTA). NIST SP 800-207. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf.
36: Edge Computing: Next Steps in Architecture, Design and Testing. URL: https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing/.
37: CYBER; Trusted Cross-Domain Interface: Interface to offload sensitive functions to a trusted domain. ETSI TS 103 457, October 2018. URL: https://www.etsi.org/deliver/etsi_ts/103400_103499/103457/01.01.01_60/ts_103457v010101p.pdf.
38: Cloud infrastructure reference model. GSMA PRD NG.126 v3.0, 2022.
39: Openstack documentation. URL: https://docs.openstack.org/.
40: Network functions virtualisation (nfv); infrastructure overview. ETSI GS NFV-INF 001 V1.1.1, January 2015. URL: https://www.etsi.org/deliver/etsi_gs/NFV-INF/001_099/001/01.01.01_60/gs_NFV-INF001v010101p.pdf.
41: Openstack use cases. URL: https://docs.openstack.org/arch-design/use-cases.html.
42: Open vswitch. URL: https://www.openvswitch.org/.
43: OpenStack Community. Openstack wallaby projects. URL: https://docs.openstack.org/wallaby/projects.html.
44: Scott O. Bradner. Key words for use in rfcs to indicate requirement levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
45: H. Philip White. Center for internet security - password policy guide. 2020. URL: https://www.cisecurity.org/insights/white-papers/cis-password-policy-guide.
46: Center for internet security - controls v7.1. URL: https://www.cisecurity.org/controls/cis-controls-list.
47: Openstack - cpu dedicated set. URL: https://docs.openstack.org/nova/latest/configuration/config.html#compute.cpu_dedicated_set.
48: Openstack - cpu topologies. URL: https://docs.openstack.org/nova/latest/admin/cpu-topologies.html.
49: Openstack - neutron plugins and drivers. URL: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers.
50: Openstack - tags. URL: https://specs.openstack.org/openstack/api-wg/guidelines/tags.html.
51: Openstack - configuring the stateful services. URL: https://docs.openstack.org/ha-guide/control-plane-stateful.html.
52: Openstack - senlin documentation. URL: https://docs.openstack.org/senlin/wallaby/.
53: Openstack - neutron ovs agent support for baremetal with smart nic. URL: https://specs.openstack.org/openstack/neutron-specs/specs/stein/neutron-ovs-agent-support-baremetal-with-smart-nic.html.
54: National telecommunications and information administration - software bill of materials. URL: https://ntia.gov/SBOM.
55: Center for internet security. URL: https://www.cisecurity.org/.
56: Cloud security alliance. URL: https://cloudsecurityalliance.org/.
57: Open web application security project - cheat sheet series. URL: https://github.com/OWASP/CheatSheetSeries.
58: Open web application security project. URL: https://www.owasp.org.
59: Open web application security project - top ten security risks. URL: https://owasp.org/www-project-top-ten/.
60: Open web application security project - software maturity model (samm). URL: https://owaspsamm.org/blog/2019/12/20/version2-community-release/.
61: Open web application security project - web security testing guide. URL: https://github.com/OWASP/wstg/tree/master/document.
62: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 27001:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en.
63: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 27002:2013. 2013. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27002:ed-2:v1:en.
64: Iso (international organization for standardization) and iec (international electrotechnical commission) iso/iec 7032:2012. 2012. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en.
65: Openstack storage. URL: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html#table-openstack-storage.
66: Openstack cinder driver support matrix. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.
67: Tungsten fabric - multicloud multistack sdn. URL: https://tungsten.io.
68: Openstack glossary. URL: https://docs.openstack.org/doc-contrib-guide/common/glossary.html.
69: Openstack feature support matrix. URL: https://docs.openstack.org/nova/latest/user/support-matrix.html.
70: Openstack storage architecture design. URL: https://docs.openstack.org/arch-design/design-storage.html.
71: Openstack nova: kvm. URL: https://docs.openstack.org/nova/wallaby/admin/configuration/hypervisor-kvm.html.
72: Openstack - hardening the virtualization layers. URL: https://docs.openstack.org/security-guide/compute/hardening-the-virtualization-layers.html.
73: Openstack reference architecture for 100, 300 and 500 nodes. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html.
74: (dpdk) release notes. URL: http://doc.dpdk.org/guides/rel_notes.
75: (dpdk) performance reports. URL: http://core.dpdk.org/perf-reports/.
76: Robert Moskowitz, Daniel Karrenberg, Yakov Rekhter, Eliot Lear, and Geert Jan de Groot. Address allocation for private internets. RFC 1918, February 1996. URL: https://www.rfc-editor.org/info/rfc1918, doi:10.17487/RFC1918.
77: Openstack - introducing octavia. URL: https://docs.openstack.org/octavia/latest/reference/introduction.html.
78: Openstack octavia (load-balancer service). URL: https://governance.openstack.org/tc/reference/projects/octavia.html.
79: Openstack/neutron-vpnaas. URL: https://opendev.org/openstack/neutron-vpnaas.
80: Openstack neutron: plugins. URL: https://wiki.openstack.org/wiki/Neutron#Plugins.
81: Openstack neutron: api extensions. URL: https://docs.openstack.org/neutron/latest/contributor/internals/api_extensions.html.
82: Openstack networking api v2.0: list extensions. URL: https://docs.openstack.org/api-ref/network/v2/#list-extensions.
83: Openstack networking api v2.0: show extension details. URL: https://docs.openstack.org/api-ref/network/v2/#show-extension-details.
84: Openstack neutron/ml2. URL: https://wiki.openstack.org/wiki/Neutron/ML2.
85: Openstack cinder driver support matrix. URL: https://docs.openstack.org/cinder/latest/reference/support-matrix.html.
86: Openstack (cinder) available drivers. URL: https://docs.openstack.org/cinder/latest/drivers.html.
87: Openstack cinder service configuration. URL: https://docs.openstack.org/cinder/latest/configuration/index.html.
88: Openstack cinder administration. URL: https://docs.openstack.org/cinder/latest/admin/index.html.
89: Ceph - the future of storage. URL: https://ceph.io/en.
90: Keystone, the openstack identity service. URL: https://docs.openstack.org/keystone/wallaby/.
91: Openstack - welcome to glance's documentation! URL: https://docs.openstack.org/glance/wallaby/.
92: Openstack block storage (cinder) documentation. URL: https://docs.openstack.org/cinder/wallaby/.
93: Openstack - welcome to swift's documentation! URL: https://docs.openstack.org/swift/wallaby/.
94: Openstack - welcome to neutron's documentation! URL: https://docs.openstack.org/neutron/wallaby/.
95: Openstack - scenario: high availability using distributed virtual routing (dvr). URL: https://docs.openstack.org/liberty/networking-guide/scenario-dvr-ovs.html.
96: Openstack neutron: distributed virtual routing with vrrp. URL: https://docs.openstack.org/neutron/wallaby/admin/config-dvr-ha-snat.html.
97: Openstack compute (nova). URL: https://docs.openstack.org/nova/wallaby/.
98: Openstack - welcome to ironic's documentation! URL: https://docs.openstack.org/ironic/wallaby/.
99: Openstack ironic api reference: bare metal api. URL: https://docs.openstack.org/api-ref/baremetal/.
100: Openstack - welcome to the heat documentation! URL: https://docs.openstack.org/heat/wallaby/.
101: Horizon: the openstack dashboard project. URL: https://docs.openstack.org/horizon/wallaby/.
102: Openstack - placement. URL: https://docs.openstack.org/placement/wallaby/index.html.
103: Openstack - placement: modeling with provider trees. URL: https://docs.openstack.org/placement/latest/user/provider-tree.html.
104: Openstack - placement usage. URL: https://docs.openstack.org/placement/latest/user/index.html.
105: Openstack key manager (barbican). URL: https://docs.openstack.org/barbican/wallaby/.
106: Openstack accelerator (cyborg). URL: https://docs.openstack.org/cyborg/wallaby/.
107: Openstack compute api guide 2.1.0: server concepts. URL: https://docs.openstack.org/api-guide/compute/server_concepts.html.
108: Openstack cyborg support matrix (wallaby). URL: https://docs.openstack.org/cyborg/wallaby/reference/support-matrix.html.
109: Openstack cyborg support matrix. URL: https://docs.openstack.org/cyborg/latest/reference/support-matrix.html.
110: Openstack cyborg architecture. URL: https://docs.openstack.org/cyborg/latest/user/architecture.html.
111: Openstack cyborg setup.cfg (wallaby). URL: https://opendev.org/openstack/cyborg/src/branch/stable/wallaby/setup.cfg.
112: Kubernetes. URL: https://kubernetes.io.
113: Reference architecture for kubernetes based cloud infrastructure. Anuket RA2. URL: https://cntt.readthedocs.io/projects/ra2/en/latest/index.html.
114: Openstack nova: flavors. URL: https://docs.openstack.org/nova/latest/user/flavors.html.
115: Open glossary of edge computing. URL: https://github.com/State-of-the-Edge/glossary/blob/master/edge-glossary.md.
116: Openstack - edge computing: next steps in architecture, design and testing. URL: https://www.openstack.org/use-cases/edge-computing/edge-computing-next-steps-in-architecture-design-and-testing.
117: Openstack reference architecture for 100, 300 and 500 nodes: services placement summary. URL: https://fuel-ccp.readthedocs.io/en/latest/design/ref_arch_100_nodes.html#services-placement-summary.
118: Openstack nova: image pre-caching. URL: https://docs.openstack.org/nova/latest/admin/image-caching.html#image-pre-caching.
119: Airship v2. URL: https://www.airshipit.org/.
120: Starlingx - deploy your edge cloud now. URL: https://www.starlingx.io/.
121: Openstack tripleo. URL: http://opendev.org/openstack/tripleo-common.
122: Openstack compute microversions. URL: https://docs.openstack.org/api-guide/compute/microversions.html.
123: Identity api v3. URL: https://docs.openstack.org/api-ref/identity/v3/index.html.
124: Identity api v3 extensions. URL: https://docs.openstack.org/api-ref/identity/v3-ext/.
125: Security compliance and pci-dss. URL: https://docs.openstack.org/keystone/wallaby/admin/configuration.html#security-compliance-and-pci-dss.
126: Image service api. URL: https://docs.openstack.org/api-ref/image/v2/.
127: Image service versions. URL: https://docs.openstack.org/api-ref/image/versions/index.html#version-history.
128: Block storage api. URL: https://docs.openstack.org/api-ref/block-storage/.
129: Cinder rest api version history. URL: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html.
130: Object storage api. URL: https://docs.openstack.org/api-ref/object-store/index.html.
131: Discoverability. URL: https://docs.openstack.org/swift/latest/api/discoverability.html.
132: Networking service apis. URL: https://docs.openstack.org/api-ref/network/.
133: Networking api v2.0. URL: https://docs.openstack.org/api-ref/network/v2/.
134: Compute api. URL: https://docs.openstack.org/api-ref/compute/.
135: Compute rest api version history. URL: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html.
136: Placement api. URL: https://docs.openstack.org/api-ref/placement/.
137: Placement rest api version history. URL: https://docs.openstack.org/placement/latest/placement-api-microversion-history.html.
138: Orchestration service api. URL: https://docs.openstack.org/api-ref/orchestration/.
139: Template version history. URL: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html.
140: Heat orchestration template (hot) specification. URL: https://docs.openstack.org/heat/latest/template_guide/hot_spec.html#rocky.
141: Openstack apis. URL: https://docs.openstack.org/api-ref/.
142: Kubernetes apis. URL: https://kubernetes.io/docs/concepts/overview/kubernetes-api/.
143: Kvm apis. URL: https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt.
144: Libvirt apis. URL: https://libvirt.org/html/index.html.
145: Barbican api. URL: https://docs.openstack.org/barbican/latest/api/.
146: Openstack - security boundaries and threats. URL: https://docs.openstack.org/security-guide/introduction/security-boundaries-and-threats.html.
147: Openstack security guide. URL: https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html.
148: Mitre - common vulnerabilities and exposures. URL: https://cve.mitre.org/.
149: National institute of standards and technology vulnerabilities metrics. URL: https://nvd.nist.gov/vuln-metrics/cvss.
150: Openstack security guide - identity. URL: https://docs.openstack.org/security-guide/identity.html.
151: Openstack security guide - authentication methods. URL: https://docs.openstack.org/security-guide/identity/authentication-methods.html.
152: Openstack security guide - policies. URL: https://docs.openstack.org/security-guide/identity/policies.html#policy-section.
153: Openstack keystone default roles. URL: https://docs.openstack.org/keystone/latest/admin/service-api-protection.html.
154: Openstack - introduction to tls and ssl. URL: https://docs.openstack.org/security-guide/secure-communication/introduction-to-ssl-and-tls.html.
155: Center for internet security cis-cat pro. URL: https://www.cisecurity.org/cybersecurity-tools/cis-cat-pro/.
156: Center for internet security benchmarks. URL: https://www.cisecurity.org/cis-benchmarks/.
157: Openstack image signature verification. URL: https://docs.openstack.org/glance/wallaby/user/signature.html.
158: Openstack - sr-iov passthrough for networking. URL: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking.
159: Openstack trusted images. URL: https://docs.openstack.org/security-guide/instance-management/security-services-for-instances.html#trusted-images.
160: Openstack virtual machine image guide. URL: https://docs.openstack.org/image-guide/.
161: Adding signed images. URL: https://docs.openstack.org/operations-guide/ops-user-facing-operations.html#adding-signed-images.
162: Network functions virtualisation (nfv) release 4; protocols and data models; vnf package and pnfd archive specification. ETSI GS NFV-SOL 004 V4.3.1, July 2022. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/004/04.03.01_60/gs_NFV-SOL004v040301p.pdf.
163: Network functions virtualisation (nfv) release 2; security; vnf package security specification. ETSI GS NFV-SEC 021 V2.6.1, June 2019. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SEC/001_099/021/02.06.01_60/gs_nfv-sec021v020601p.pdf.
164: Foreman. URL: https://www.theforeman.org/.
165: Ansible documentation. URL: https://docs.ansible.com/.
166: Openstack tripleo architecture. URL: https://docs.openstack.org/tripleo-docs/latest/install/introduction/architecture.html#project-architecture.
167: Ovp. URL: https://www.opnfv.org/verification.
168: Jenkins. URL: https://build.opnfv.org/.
169: Test dabase. URL: https://docs.opnfv.org/en/stable-hunter/_images/OPNFV_testing_working_group.png.
170: S3 compatible storage service. URL: http://artifacts.opnfv.org/.
171: Functest-wallaby-zip. URL: https://build.opnfv.org/ci/job/functest-wallaby-zip/4/console.
172: Xtesting ci. URL: https://galaxy.ansible.com/collivier/xtesting.
173: Docker. URL: https://www.docker.com/.
174: Xtesting. URL: https://xtesting.readthedocs.io/en/latest/.
175: Opnfv fraser. URL: https://www.sdxcentral.com/articles/news/opnfvs-6th-release-brings-testing-capabilities-that-orange-is-already-using/2018/05/.
176: Xtesting python package. URL: https://pypi.org/project/xtesting/.
177: Test case execution description. URL: https://git.opnfv.org/functest-xtesting/tree/docker/core/testcases.yaml.
178: Ci/cd toolchains in a few commands. URL: https://github.com/collivier/ansible-role-xtesting#readme.
179: Ci/cd deployment models. URL: https://lists.opnfv.org/g/opnfv-tsc/message/5702.
180: Anuket releng. URL: https://git.opnfv.org/releng/tree/jjb/functest.
181: Test case result dump. URL: http://artifacts.opnfv.org/functest/9ID39XK47PMZ.zip.
182: Xtesting samples. URL: https://git.opnfv.org/functest-xtesting/plain/ansible/site.yml?h=stable/wallaby.
183: Openstack verification. URL: https://git.opnfv.org/functest/plain/ansible/site.yml?h=stable/wallaby.
184: Anuket rc1. URL: https://git.opnfv.org/functest/plain/ansible/site.cntt.yml?h=stable/wallaby.
185: Kubernetes verification. URL: https://git.opnfv.org/functest-kubernetes/plain/ansible/site.yml?h=stable/v1.22.
186: Functest. URL: https://functest.readthedocs.io/en/stable-wallaby/.
187: Refstack. URL: https://refstack.openstack.org/.
188: Networking bgpvpn. URL: https://docs.openstack.org/networking-bgpvpn/latest/.
189: Networking sfc. URL: https://docs.openstack.org/networking-sfc/latest/.
190: Devstack gates. URL: https://docs.opendev.org/opendev/system-config/latest/devstack-gate.html.
191: Rally. URL: https://github.com/openstack/rally-openstack.
192: Temptest. URL: https://github.com/openstack/tempest.
193: Temptest. URL: https://docs.openstack.org/devstack/latest/.
194: Raspberry pi. URL: https://www.raspberrypi.org/.
195: Functest daily jobs. URL: https://build.opnfv.org/ci/view/functest/job/functest-wallaby-daily/17/.
196: Openstack performance tools. URL: https://docs.openstack.org/developer/performance-docs/methodologies/tools.html.
197: Run alpine functest containers (wallaby). URL: https://wiki.anuket.io/display/HOME/Functest+Wallaby.
198: Functest gates. URL: https://build.opnfv.org/ci/view/functest.
199: New functest cntt containers. URL: https://lists.opnfv.org/g/opnfv-tsc/message/5717.
200: Keystone-tempest-plugin. URL: https://opendev.org/openstack/keystone-tempest-plugin.
201: Functest smoke cntt. URL: https://git.opnfv.org/functest/tree/docker/smoke-cntt/testcases.yaml?h=stable%2Fwallaby.
202: Cinder-tempest-plugin. URL: https://opendev.org/openstack/cinder-tempest-plugin.
203: Functest review 68881. URL: https://gerrit.opnfv.org/gerrit/68881.
204: Functest review 71011. URL: https://gerrit.opnfv.org/gerrit/71011.
205: Openstack bug 1770179. URL: https://launchpad.net/bugs/1770179.
206: Openstack bug 1677525. URL: https://launchpad.net/bugs/1677525.
207: Openstack bug 1317133. URL: https://launchpad.net/bugs/1317133.
208: Openstack bug 1905432. URL: https://launchpad.net/bugs/1905432.
209: Neutron-tempest-plugin. URL: https://opendev.org/openstack/neutron-tempest-plugin.
210: Openstack bug 1863707. URL: https://launchpad.net/bugs/1863707.
211: Functest review 69105. URL: https://gerrit.opnfv.org/gerrit/69105.
212: Openstack bug 1676207. URL: https://launchpad.net/bugs/1676207.
213: Openstack bug 1836595. URL: https://launchpad.net/bugs/1836595.
214: Openstack bug 1186354. URL: https://launchpad.net/bugs/1186354.
215: Openstack bug 1014647. URL: https://launchpad.net/bugs/1014647.
216: Openstack bug 1311500. URL: https://launchpad.net/bugs/1311500.
217: Openstack bug 1161411. URL: https://launchpad.net/bugs/1161411.
218: Openstack bug 1540645. URL: https://launchpad.net/bugs/1540645.
219: Heat-tempest-plugin. URL: https://opendev.org/openstack/heat-tempest-plugin.
220: Openstack story 2007804. URL: https://storyboard.openstack.org/#!/story/2007804.
221: Functest review 69926. URL: https://gerrit.opnfv.org/gerrit/69926.
222: Functest review 69931. URL: https://gerrit.opnfv.org/gerrit/69931.
223: Functest review 70004. URL: https://gerrit.opnfv.org/gerrit/70004.
224: Tempest-horizon. URL: https://github.com/openstack/tempest-horizon.
225: Functest healthcheck. URL: https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable%2Fwallaby.
226: Functest benchmarking cntt. URL: https://git.opnfv.org/functest/tree/docker/benchmarking-cntt/testcases.yaml?h=stable%2Fwallaby.
227: Rally_full_cntt. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally_full_cntt-run-5/rally_full_cntt/rally_full_cntt.html.
228: Rally_jobs_cntt. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-cntt-wallaby-rally_jobs_cntt-run-5/rally_jobs_cntt/rally_jobs_cntt.html.
229: Vmtp. URL: http://vmtp.readthedocs.io/en/latest.
230: Shaker. URL: https://pyshaker.readthedocs.io/en/latest/.
231: Vmtp scenarios. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.json.
232: Functest vmtp. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-vmtp-run-8/vmtp/vmtp.html.
233: Shaker scenarios. URL: http://artifacts.opnfv.org/functest/KDBNITEN317M/functest-opnfv-functest-benchmarking-wallaby-shaker-run-8/shaker/report.json.
234: Functest vnf. URL: https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable%2Fwallaby.
235: Clearwater ims. URL: https://clearwater.readthedocs.io/en/stable/.
236: Vyos vrouter. URL: https://www.vyos.io/.
237: Openairinterface vepc. URL: https://www.openairinterface.org/.
238: Cloudify. URL: https://cloudify.co.
239: Juju. URL: https://jaas.ai/.
240: Clearwater-live-test. URL: https://github.com/Metaswitch/clearwater-live-test.
241: Docker http/https proxy. URL: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy.
242: Openstack autoscaling with heat. URL: https://docs.openstack.org/senlin/latest/scenarios/autoscaling_heat.html.
243: Reference model for cloud infrastructure (rm). GSMA PRD NG.126 v3.0, 2022.
244: Kubernetes documentation. URL: https://kubernetes.io/docs/home/.
245: What is kubernetes. URL: https://kubernetes.io/docs/concepts/overview/.
246: Scott O. Bradner. Key words for use in rfcs to indicate requirement levels. RFC 2119, March 1997. URL: https://www.rfc-editor.org/info/rfc2119, doi:10.17487/RFC2119.
247: Cis password policy guide. URL: https://www.cisecurity.org/insights/white-papers/cis-password-policy-guide.
248: Cis controls list. URL: https://www.cisecurity.org/controls/cis-controls-list.
249: Cve - common vulnerabilities and exposures. URL: https://cve.mitre.org/.
250: Sbom - software bill of materials. URL: https://ntia.gov/page/software-bill-materials.
251: Cloud security alliance. URL: https://cloudsecurityalliance.org/.
252: Owasp cheat sheet series (ocss). URL: https://github.com/OWASP/CheatSheetSeries/.
253: Owasp top ten security risks. URL: https://owasp.org/www-project-top-ten/.
254: Owasp software maturity model (samm). URL: https://owaspsamm.org/.
255: Owasp web security testing guide. URL: https://github.com/OWASP/wstg/.
256: Iso/iec 27001. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27001:ed-2:v1:en.
257: Iso/iec 27032. URL: https://www.iso.org/obp/ui/#iso:std:iso-iec:27032:ed-1:v1:en.
258: Cncf kubernetes conformance. URL: https://github.com/cncf/k8s-conformance.
259: Open container initiative (oci) runtime spec. URL: https://github.com/opencontainers/runtime-spec.
260: Kubernetes docs - pod. URL: https://kubernetes.io/docs/concepts/workloads/pods.
261: Kubernetes docs - replicaset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset.
262: Kubernetes docs - deployment. URL: https://kubernetes.io/docs/concepts/workloads/controllers/deployment.
263: Kubernetes docs - daemonset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset.
264: Kubernetes docs - job. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job.
265: Kubernetes docs - cronjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs.
266: Kubernetes docs - statefulset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset.
267: Kubernetes docs - cpu manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies.
268: Kubernetes docs - huge pages. URL: https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages.
269: Kubernetes docs - memory manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager.
270: Kubernetes docs - topology manager. URL: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager.
271: Kubernetes node feature discovery. URL: https://kubernetes-sigs.github.io/node-feature-discovery/.
272: Kubernetes docs - device plugin framework. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.
273: Kubernetes sr-iov network device plugin. URL: https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin.
274: Github: multus-cni. URL: https://github.com/k8snetworkplumbingwg/multus-cni.
275: Kubernetes docs - network plugins. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins.
276: Kubernetes network custom resource definition. URL: https://github.com/k8snetworkplumbingwg/multi-net-spec.
277: Tungsten fabric. URL: https://tungsten.io/.
278: Af_xdp device plugin. URL: https://github.com/intel/afxdp-plugins-for-kubernetes.
279: Cloud native data plane. URL: https://cndp.io/.
280: Dpdk af_xdp poll mode driver. URL: https://doc.dpdk.org/guides/nics/af_xdp.html.
281: Kubernetes docs - ingress. URL: https://kubernetes.io/docs/concepts/services-networking/ingress.
282: Kubernetes docs - service. URL: https://kubernetes.io/docs/concepts/services-networking/service.
283: Kubernetes docs - endpointslices. URL: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices.
284: Kubernetes docs - network policies. URL: https://kubernetes.io/docs/concepts/services-networking/network-policies.
285: Container network interface. URL: https://github.com/containernetworking/cni.
286: Kubernetes network special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-network.
287: Kubernetes network plumbing working group. URL: https://github.com/k8snetworkplumbingwg/community.
288: Helm documentation. URL: https://helm.sh/docs.
289: Kubernetes docs - custom resources. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources.
290: Kubernetes docs - custom resource definitions. URL: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.
291: Kubernetes docs - api server aggregation. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation.
292: Kubernetes docs - custom controllers. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-controllers.
293: Kubernetes docs - operator pattern. URL: https://kubernetes.io/docs/concepts/extend-kubernetes/operator.
294: Operator hub. URL: https://operatorhub.io.
295: Reference implementation based on ra2 specifications (ri2). URL: https://cntt.readthedocs.io/projects/ri2/.
296: Reference architecture (ra1) for openstack based cloud infrastructure. URL: https://cntt.readthedocs.io/projects/ra1/.
297: Node feature discovery. URL: https://kubernetes-sigs.github.io/node-feature-discovery/stable/get-started/index.html.
298: Kubernetes distributions and platforms document. URL: https://docs.google.com/spreadsheets/d/1uF9BoDzzisHSQemXHIKegMhuythuq_GL3N1mlUUK2h0/.
299: Kubernetes supported versions. URL: https://kubernetes.io/releases/version-skew-policy/#supported-versions.
300: Kubernetes alpha api. URL: https://kubernetes.io/docs/reference/using-api/#api-versioning.
301: Kubernetes publishing services (servicetypes). URL: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types.
302: Kubernetes ingress. URL: https://kubernetes.io/docs/concepts/services-networking/ingress/.
303: Open container initiative runtime specification, version 1.0.0. URL: https://github.com/opencontainers/runtime-spec/blob/v1.0.0/spec.md.
304: Kubernetes: introducing container runtime interface (cri) in kubernetes. URL: https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/.
305: Kubernetes network custom resource definition de-facto standard. URL: https://github.com/k8snetworkplumbingwg/multi-net-spec/tree/master/v1.2.
306: Kubernetes csi drivers. URL: https://kubernetes-csi.github.io/docs/drivers.html.
307: Application service descriptor (asd) and packaging proposals for cnf. URL: https://wiki.onap.org/display/DW/Application+Service+Descriptor+%28ASD%29+and+packaging+Proposals+for+CNF.
308: Etsi gs nfv-sol 001: nfv descriptors based on tosca specification. ETSI GS NFV-SOL 001, 2022. URL: https://www.etsi.org/deliver/etsi_gs/NFV-SOL/001_099/001/04.02.01_60/gs_NFV-SOL001v040201p.pdf.
309: Kubernetes types of volumes, hostpath. URL: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath.
310: Cnf testsuite, rationale, test if the helm chart is published: helm_chart_published. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-helm-chart-is-published-helm_chart_published.
311: Cnf testsuite, rationale, test if the helm chart is valid: helm_chart_valid. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-helm-chart-is-valid-helm_chart_valid.
312: Cnf testsuite, rationale, to test if the cnf can perform a rolling update: rolling_update. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-the-cnf-can-perform-a-rolling-update-rolling_update.
313: Cnf testsuite, rationale, to check if a cnf version can be downgraded through a rolling_downgrade: rolling_downgrade. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-a-cnf-version-can-be-downgraded-through-a-rolling_downgrade-rolling_downgrade.
314: Cnf testsuite, rationale, to check if the cnf is compatible with different cnis: cni_compatibility. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-is-compatible-with-different-cnis-cni_compatibility.
315: Cnf testsuite, rationale, test if the cnf crashes when node drain occurs: node_drain. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-node-drain-occurs-node_drain.
316: Cnf testsuite, rationale, test if the cnf crashes when network latency occurs: pod_network_latency. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-network-latency-occurs-pod_network_latency.
317: Cnf testsuite, rationale, test if the cnf crashes when disk fill occurs: disk_fill. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-disk-fill-occurs-disk_fill.
318: Cnf testsuite, rationale, test if the cnf crashes when pod memory hog occurs: pod_memory_hog. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-memory-hog-occurs-pod_memory_hog.
319: Cnf testsuite, rationale, test if the cnf crashes when pod io stress occurs: pod_io_stress. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-io-stress-occurs-pod_io_stress.
320: Cnf testsuite, rationale, test if the cnf crashes when pod network corruption occurs: pod_network_corruption. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-corruption-occurs-pod_network_corruption.
321: Cnf testsuite, rationale, test if the cnf crashes when pod network duplication occurs: pod_network_duplication. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-duplication-occurs-pod_network_duplication.
322: Cnf testsuite, rationale, to test if the cnf uses local storage: no_local_volume_configuration. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-the-cnf-uses-local-storage-no_local_volume_configuration.
323: Cnf testsuite, rationale, to test if there is a liveness entry in the helm chart: liveness. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-is-a-liveness-entry-in-the-helm-chart-liveness.
324: Cnf testsuite, rationale, to test if there is a readiness entry in the helm chart: readiness. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-is-a-readiness-entry-in-the-helm-chart-readiness.
325: Cnf testsuite, rationale, to check if there is automatic mapping of service accounts: service_account_mapping. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-is-automatic-mapping-of-service-accounts-service_account_mapping.
326: Cnf testsuite, rationale, to check if there is a host network attached to a pod: host_network. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-is-a-host-network-attached-to-a-pod-host_network.
327: Cnf testsuite, rationale, to check if containers are running with hostpid or hostipc privileges: host_pid_ipc_privileges. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-are-running-with-hostpid-or-hostipc-privileges-host_pid_ipc_privileges.
328: Cnf testsuite, rationale, to check if containers have resource limits defined: resource_policies. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-have-resource-limits-defined-resource_policies.
329: Cnf testsuite, rationale, to check if containers have immutable file systems: immutable_file_systems. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-have-immutable-file-systems-immutable_file_systems.
330: Kubernetes documentation: images. URL: https://kubernetes.io/docs/concepts/containers/images/.
331: Cnf testsuite, rationale, to test if there are any (non-declarative) hardcoded ip addresses or subnet masks in the k8s runtime configuration: hardcoded_ip_addresses_in_k8s_runtime_configuration. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-are-any-non-declarative-hardcoded-ip-addresses-or-subnet-masks-in-the-k8s-runtime-configuration-hardcoded_ip_addresses_in_k8s_runtime_configuration.
332: Kubernetes documentation: service. URL: https://kubernetes.io/docs/concepts/services-networking/service/.
333: Kubernetes documentation: immutable configmaps. URL: https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable.
334: Kubernetes documentation: horizontal pod autoscaling. URL: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/.
335: Cnf testsuite, rationale, to check if the cnf has a reasonable image size: reasonable_image_size. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-has-a-reasonable-image-size-reasonable_image_size.
336: Cnf testsuite, rationale, to check if the cnf have a reasonable startup time: reasonable_startup_time. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-the-cnf-have-a-reasonable-startup-time-reasonable_startup_time.
337: Cnf testsuite, rationale, to check if there are any privileged containers: privileged_containers. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-there-are-any-privileged-containers-privileged_containers.
338: Cnf testsuite, rationale, to check if any containers are running as a root user (checks the user outside the container that is running dockerd): non_root_user. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-any-containers-are-running-as-a-root-user-checks-the-user-outside-the-container-that-is-running-dockerd-non_root_user.
339: Cnf testsuite, rationale, to check if any containers allow for privilege escalation: privilege_escalation. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-any-containers-allow-for-privilege-escalation-privilege_escalation.
340: Cnf testsuite, rationale, to check if containers are running with non-root user with non-root membership: non_root_containers. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-check-if-containers-are-running-with-non-root-user-with-non-root-membership-non_root_containers.
341: Kubernetes documentation: recommended labels. URL: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/.
342: The twelve factor app: logs. URL: https://12factor.net/logs.
343: Cnf testsuite, rationale, to test if there are host ports used in the service configuration: hostport_not_used. URL: https://github.com/cncf/cnf-testsuite/blob/main/RATIONALE.md#to-test-if-there-are-host-ports-used-in-the-service-configuration-hostport_not_used.
344: Kubernetes documentation ports and protocols. URL: https://kubernetes.io/docs/reference/networking/ports-and-protocols/.
345: Building secure microservices-based applications using service-mesh architecture. NIST Special Publication 800-204A, 2020. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-204a.pdf.
346: Attribute-based access control for microservices-based applications using a service mesh. NIST Special Publication 800-204B, 2020. URL: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-204B.pdf.
347: Arstechnica: tesla cloud resources are hacked to run cryptocurrency-mining malware. URL: https://arstechnica.com/information-technology/2018/02/tesla-cloud-resources-are-hacked-to-run-cryptocurrency-mining-malware/.
348: Kubernetes testing special interest group. URL: https://github.com/kubernetes/community/blob/master/sig-testing/charter.md.
349: Kubernetes end-to-end testing. URL: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md.
350: Kubernetes feature gates. URL: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates.
351: Kubernetes kep-3136. URL: https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/3136-beta-apis-off-by-default/README.md.
352: Kubernetes api. URL: https://kubernetes.io/docs/reference/using-api/.
353: Kubernetes api reference. URL: https://kubernetes.io/docs/reference/kubernetes-api/.
354: Kubernetes api groups. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/ #-strong-api-groups-strong-.
355: Kubernetes api machinery special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-api-machinery.
356: Kubernetes feature crossnamespacepodaffinity. URL: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#namespace-selector.
357: Kubernetes feature storageversionapi. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#storageversion-v1alpha1-internal-apiserver-k8s-io.
358: Kubernetes apps special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-apps.
359: Kubernetes feature daemonsetupdatesurge. URL: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#rollingupdatedaemonset-v1-apps.
360: Kubernetes feature indexedjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job/.
361: Kubernetes feature statefulset. URL: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/.
362: Kubernetes feature suspendjob. URL: https://kubernetes.io/docs/concepts/workloads/controllers/job/#suspending-a-job.
363: Kubernetes feature tainteviction. URL: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions.
364: Kubernetes feature ttlafterfinished. URL: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/.
365: Kubernetes auth special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-auth.
366: Kubernetes feature boundserviceaccounttokenvolume. URL: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/1205-bound-service-account-tokens/README.md.
367: Kubernetes cluster lifecycle special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle.
368: Kubernetes instrumentation special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-instrumentation.
369: Kubernetes network special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-network.
370: Kubernetes feature ipv6dualstack. URL: https://kubernetes.io/docs/concepts/services-networking/dual-stack/.
371: Kubernetes node special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-node.
372: Kubernetes feature probeterminationgraceperiod. URL: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#probe-level-terminationgraceperiodseconds.
373: Kubernetes feature downwardapihugepages. URL: https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information.
374: Kubernetes feature podreadinessgate. URL: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-readiness-gate.
375: Kubernetes feature sysctls. URL: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/.
376: Kubernetes scheduling special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-scheduling.
377: Kubernetes feature localstoragecapacityisolation. URL: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.
378: Kubernetes storage special interest group. URL: https://github.com/kubernetes/community/tree/master/sig-storage.
379: Alibaba cloud blog: what can we learn from twitter's move to kubernetes. URL: https://www.alibabacloud.com/blog/what-can-we-learn-from-twitters-move-to-kubernetes_595156.
380: Youtube: kubernetes failure stories, or: how to crash your cluster - henning jacobs. URL: https://www.youtube.com/watch?v=LpFApeaGv7A.
381: Cncf blog: demystifying kubernetes as a service – how alibaba cloud manages 10,000s of kubernetes clusters. URL: https://www.cncf.io/blog/2019/12/12/demystifying-kubernetes-as-a-service-how-does-alibaba-cloud-manage-10000s-of-kubernetes-clusters/.
382: Google docs: kep: multinetwork podnetwork object. URL: https://docs.google.com/document/d/17LhyXsEgjNQ0NWtvqvtgJwVqdJWreizsgAZHWflgP-A/edit.
383: Kubernetes docs: user namespaces. URL: https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/.
384: Kep-127: support user namespaces in stateless pods. URL: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/127-user-namespaces.
385: Wikipedia: linux namespaces. URL: https://en.wikipedia.org/wiki/Linux_namespaces.