TSD Security Handbook

1. TSD information security policy

TSD is a special purpose eInfrastructure for sensitive data. It is a multi-tenant plaform-as-a-service (PAAS), offering secure login, remote access to virtual machines, APIs for application development, and user support. This is offered to customers such as research projects, clinics, students, centers of excellence, private companies and more. TSD processes data on behalf of its customers based on the establishment of data processing agreements, and commercial agreements. Customers in turn, as data owners, are responsible for establishing their own legal basis for data processing, approved by recognised ethical approval bodies such as data protection officers.

TSD develops a set of core research services and APIs in-house, and consumes infrastructure services from other parts of the IT-department at UiO. Other parts of the organistion, in turn, consume TSD APIs in order to build research services according to user needs.

As a data processor, TSD is commmitted to ensuring data confidentiality, in tegrity, and availability of all data processed by the service. This includes data that is being transmitted to and from the service, while using TSD’s data transport services. Confidentiality applies to users, and staff. Integrity applies to all processed data. Availability applies to all services, given the published service level agreement. Remaining details are specified in topic-specific policies.

2. Topic-specific policies

2.1 Defense in depth

Security mechanisms are applied at all levels of the technology stack (physical, virtual, network, IP, application layer)

2.2 Tenant separation

TSD project tentants shall not have any access to data or services running in other TSD projects by default, any such access shall be controlled by TSD and set up on approval by principle investigators in both projects

2.3 Data minimisation

TSD as a service provider shall only collect Personally identifiable information (PII) to the extent necessary for effective service delivery. Customers shall be encouraged by TSD to apply the principle of data minimisation in their projects. Services which integrate with TSD APIs shall be encouraged to apply data minimisation in their design and implementation, and be configured with access rights which support this principle.

2.4 Privacy by design

System architecture, software architecture, and service design shall consider data privacy by design.

2.5 Authentication with highest level of assurance possible

TSD’s own Identity Provider shall prompt users for authentication with the highest level of assurance possible, such as Two-Factor Authentication for all login. API clients that need machine-to-machine integrations for application shall be secured by additional network mechanisms.

2.6 Zero-trust remote access

TSD user can log in to virtual machines from anywhere in the world, on any network, with any device. TSD is designed such that remote access clients need not be trusted.

2.7 Mandatory access control

TSD project implement a set of mandatory access control policies by default:

• no internet access from virtual machines

• project admins are assigned by the principal investigator

• project admins, once assigned, can assign other project admins • user creation in projects must be approved by project admins • export rights for project members must be explicitly granted by the princi

pal investigator (PI) or by project admins delegated by the PI • virtual machine access must be explicitly granted

• no access to other projects (data, networks, services)

• no root access to virtual machines

• sudo access is given per command, per user, per virtual machine • when a project has ended, all network access is immediately revoked

API access control is implemented by TSD, and is deny-first. API clients are given access:

• for a limited time period of time

• per project

• per API

2.8 Discretionary access control

Principal investigators (who are admins by default) can assign project admins, and project admins can assign other admins. Project admins can grant and revoke a set of default rights. TSD operators do not perform such access control setting on behalf of users.

2.10 Access control audit

All changes to access control generate audit events with the following information:

• who performed the action

• when it was performed

• which right was granted

2.12 Secure software development

The following risks should be mitigated.

Risk: logical bugs cause security incidents

• all software should be kept in version control, allowing for easy rollback • software should be unit tested

• software should have integration tests, if relevant

• all changes should be applied via pull requests

• all pull request are subject to code review by at least one other staff member

• the use of AI code assistants should be disclosed in the pull request, so reviewers can ensure that the code author understands the proposed change - this is especially important for high risk changes, and for changes to core business logic

• conformance to style guides to ease code review

• branch protection to avoid bypassing code review

• all tests should pass before review approval

Risk: code changes applied by attacker cause security incidents

• commits should be signed with GPG keys

• access control on repositories - principle of least privilege • the use of AI code assistants should be disclosed - AI generated code could insert malicious code

Risk: secrets leaked in source code

• no secrets stored in code

• secrets should be stored in vault

• if secrets cannot be stored in vault, then file system access control should limit access

• secrets should be secured such that they are not visible in PID listings

Risk: PII information is leaked during development • no PII information in tests or in test environment

Risk: malicious dependency causes security incidents

• changes to dependencies are made in separate pull requests (version in crease and/or new dependencies) - the developer should read the relevant changelogs

• dependencies should if possible be fetched from a source where the down loaded content for the used version is verified using checksums or encryption signatures.

• vulnerability scanning on all dependencies prior to build

• all tests should pass before build

• build artefacts should be published in approved artefactories, and pulled into prod from there

• the use of AI code assistants should be disclosed - AI assistants are dependencies, and this should be made explicit

2.13 Principle of least privilege

Whenever an operator is given more access, the minimal amount of access needed to accomplish the task shall be given, for the time period necessary.

2.14 Transparency from suppliers

Suppliers who deliver infrastucture services to TSD from inside the USIT organ isation shall report production incidents, coordinate downtime, document their solutions, and have regular meetings with TSD leadership.

3. Information classification

System Approved classification

Email Yellow

Mattermost Red

RT Yellow

github.uio Green

github.com Green

ssh/ops Black

4. Password policy

4.1 Password change frequency

• All users must change their passwords at least once every year. • The first mandatory change will be required in 2024-01-04. • Subsequent changes must be made every year thereafter.

4.2 Password requirements

• Guidelines for creating strong passwords

4.3 Password change process

• Users will receive a notification via email 30 days before their password is due for renewal.

• Password changes can be made through TSD Selfservice Portal. • Users who fail to change their password by the deadline will have their accounts temporarily suspended until the password is updated.

4.4 Support and assistance

• For assistance with changing passwords or any related issues, please contact IT Help.

4.5 Compliance

• Compliance with this policy is mandatory.

Work process

TSD applies the following work process:

Figure 1: Process diagram

High level plans and priorities are set by leadership at the portfolio council, in quarterly planning meetings with team leaders, and in regular dialogue with stakeholders (IT-Security, UiO-IT leadership, service providers within UiO-IT, and customers). Such plans are recorded in the meta-plannning repository. Issues created in this repository are tagged with team-specific labels, so that they are reflected in team Kanban boards. Sometimes user support cases will necessitate significant changes to services, or significant amount of effort. In these cases, the use case is discussed at the portfolio council, prioritised, and planned.

Signficant changes to TSD are discussed in detail with IT-Security at the change council (endringsråd). The impact of applying the proposed changes are considered agains the TSD risk analysis (ROS). If a proposed change is approved, the task will be reflected the relevant team Kanban boards.

Teams are responsible for their own detailed internal work planning, given the priorities from leadership. Such plans are translated into issues on specific code repositories, or other meta planning repositories. These issues are also reflected in the team Kanban boards.

Development work is organised as pull requests. Pull requests are reviewed and tested before being merged. If a pull request has a particular relevance to a risk element in the TSD ROS, a link is provided to contextualise the change. Once merged, new artefacts are built and deployed. If a bug is discovered, an issue is created on the appropriate repository and prioritised in the team Kanban board.

If an incident occurs, due to a development or operational issue, an incident report is created in the incidents repository. Each incident has follow-up actions defined, which may or may not refer to issues on specific component repositories. Incidents are tagged with team labels, and are therefore present in the team Kanban boards. Relevance to the TSD ROS is indicated by linking to the relevant risk element.

Operations are peformed as a result of either 1) planning processes, 2) incident response, or 3) user support.

TSD organisation

Work is organised primarily in teams, each with their own leader, mandate, scope, and members. The team leader is responsible for coordinating work and plans. This has to be done in relation to plans and priorities decided by leadership. The teams are:

1. User support and stakeholder contact (label: user)

2. Web services (label: web)

3. APIs and services (label: api)

4. Devops (label: devops)

5. Infrastructure management (label: infra)

6. HPC (label: hpc)

7. Appnodes (label: appn)

Some work does not naturally belong to a single team, and is sometimes tempo rary in nature. In such cases, leadership delegates responsibility for the task, and the responsible person collaborates across teams as necessary. An example of this is TSD’s participation in external projects.

Principles

• flexibility

– team leadership and membership is always open to change, agreed upon with leadership

– the members do not limit who can participate in work

– some tasks which do not fit well into a single team are owned by an individual and driven to completion by collaborating across teams as necessary

• transparency

– it should be possible for anyone in any team to get insight into the work of other teams

• accountability

– teams take ownership of their mandate and are responsibile for fulfill ing it

Practices

• planning

– documented at a high level

– coordinated with plans from leadership

• prioritisation

– team priorities should be harmonised with:

∗ overall priorities as decided by leadership

∗ other teams

• change management

– all significant changes should be presented at the endringsråd, and approved before adoption

– a significant change can be technical or policy related

– all incidents are reported and handled in the incident review • retrospectives

– team leaders present overviews of what they have done at a common retrospective, organised by leadership

• work tracking

– work should be organised via github issues and projects

– bigger issues must be reflected in the tsd-planning repo as EPICs ∗ this ensures that everyone is informed about platform changes • communication

– team-specific mattermost channels should be documented here, and be open to anyone

– existing coordination channels

∗ TSD Core: dialy ops

∗ tsd-core-core: discussions that are too specific for TSD Core ∗ tsd-ec-dev: all PRs should be announced here, dev discussions that interest a broad audience

∗ local-hpc-team: HPC daily ops

Roles and responsibilities

• leadership

– sets priorities and high-level plans at the portfolio council – makes final decisions about platform architecture

– appoints team leaders, and members

– delegates responsibility for tasks falling outside of team scope – resolves conflicts among priorities if they arise

– responsible for ISO compliance (management system, system descrip tion, risk assessment)

• team leader

– makes detailed plans and priorities

– assigns ownership for tasks

– follow-up and task coordination (within the team and with other teams)

– reporting to leadership

– should invest in knowledge of TSD as a whole

– contribute to implementing ISO compliance

• team member

– execute tasks according to:

∗ rules and regulations

∗ implemented safety and quality procedures and instructions – keep updated on procedures

– take ownership of tasks

– drives tasks to completion

– coordinates as necessary

User support and stakeholder contact

Leader: Haneef Awan

Mandate: To organise the 2nd-line support rotation, and coordinate this effort with IT-Help. To gather intelligence from the 2nd-line support activities for improving documentation, processes, and to develop new functionality in ex isting services. Manage relationships with large customers. Collaboration and coordination with FFU-kontakt. Invoicing and contracting.

Portfolio: vakt, tsd-contact, customer relations, collaboration with Advanced User Support, invoicing, contracts, GDP and compliance

Members:

• All TSD employees, as needed

Web services

Leader: Patrick Murmann

Mandate: The development of TSD’s web services, running outside and inside of the platform. Ensuring that the web services work in harmony with each other, that functionality and help can be discovered in the right context. Interfacing with other teams such as UX and API as necessary.

Portfolio: selfservice, data portal, consent, publication, FAIR, tables, risk analysis system

Members:

• Haneef Awan

• Armen Michaeli

• Trond Thorbjørnsen

• Wojciech Olejarz

• Dmitri Darine

APIs and services

Leader: Milen Kouylekov

Mandate: The development and maintenance of TSD’s APIs, client libraries, and command-line tools. Ensure: a sound architecture, sustainable integrations, that APIs are documneted to enable integrations. The development of micro-services that integrate with the APIs. Libraries should be modular, re-used and well documented.

Portfolio: APIS, Microservices

Members:

• Benjamin Sørli Ormset

• Petter Reinholdtsen

• Eirik Haatveit

• Torjus Håkestad

• Trond Thorbjørnsen

• Fabian Bernal

• Martin Benonissen

• Haneef Awan

• Per Kulseth Dahl

Devops

Leader: Eirik Haatveit

Mandate: Ensure all teams follow best practices for development and deployment of software. Establish processes, document best practices, research and recommend adoption of new tools. Ensure TSD uses enabling services at USIT in the correct way.

Portfolio: containers, OKD, vault, github, artefactory, ACME, harbor Members:

• Fabian Bernal

• Armen Michaeli

• Torjus Håkestad

• Milen Kouylekov

• Trond Thorbjørnsen

Infrastructure management

Leader: Martin Benonissen

Mandate: That the firewall can implement security policies in a scalable way. Managing special hosts (such as fx03), and RabbitMQ. Manage the VM life-cycle, in collaboration with VDI and virt-core. Ensure proper usage of infrastructure services such as ELK, Zabbix. Oversee TSD’s usage of database services.

Portfolio: firewall, special hosts, VM life-cycle management, RabbitMQ brokers, databases, ELK, ZAbbix, DNS, Infrastructure as code

Members:

• Torjus Håkestad

• Benjamin Sørli Ormset

• Petter Reinholdtsen

• Eirik Haatveit

• Bart te Lindert

HPC

Leader: Bart te Lindert

Mandate: The hpc team shall have overall responsibility for USIT’s HPC re sources, in collaboration with relevant sections and groups.

Portfolio: fox, colossus, light-hpc, ml nodes

Members:

• Axel Rosen

• Jacob Ziemke

• Truls Mathiassen

• Ole Saastad

Appnodes

Leader: Frode Strømsvåg

Mandate: The app node team shall have overall responsibility for USIT’s app nodes, in collaboration with relevant sections and groups.

Portfolio: Special purpose research hardware

Members:

• Bart te Lindert

• Axel Rosen

• Tobias Lisbakken

• Patrick Slåttnes

• Audun Kuhne Johansen

TSD Identity and Access Management High level architecture

client -> pylib -> API proxy -> tsd-iam -> rabbitmq <- sync -> AD |

tsd-auth-api

- authentication -> sentry

- authorization

TSD account life cycle

User Creation

TSD user accounts can be created using three different methods:

1) Manually, by a TSD admin account, using the sysadmin command-line tool ur

Manual user creation is avoided as far as possible, as it necessitates managing user credentials on behalf of others. In the few cases where it must be done, TSD staff adhere to the procedures for handling authentication information (described below).

2) Using the self-service portal, with trusted 3rd-party login

Self-service user creation using trusted 3rd-party login is the preferred method, and leverages Digdir’s IdP service ID-porten (Norwegian national eID, and eIDAS). Principal investigators provide their personal information to TSD via their project application. Project applications are submitted with ID-porten login. Nettskjema get the personal number from ID-porten login, and looks up the PI’s full name in kontaktregitsteret. TSD staff review the project application. If approved, a person object is created. TSD sends confirmation of project creation to the PI. When the PI logs in to the self-service portal, they create their own user, choose their own password, and set up their own time-based one-time code (TOTP) for 2FA. Users who are not PIs are project members, and have to apply for project membership. This is done by logging in to the self-service portal with ID-porten, and registering an application for a specific project. Project admins can choose to approve the application, at which point the user is created. TSD sends a link to the user. The user logs in with ID-porten to set their credentials.

3) Using the self-service portal, with TSD’s foreign user creation protocol

For cases where researchers do not have Norwegian eIDs, or eIDAS eIDs, project admins can create user accounts in collaboration with the researchers. Project admins create short-lived, password protected invitation links. When creating the link project admins store the applicant’s name, email, date of birth, passport number, and country - information which the are responsible for collecting from the applicant themselves, via a channel not managed by TSD. TSD sends the link to the applicant’s email address. The password is shared by the project admin with the applicant via a channel not managed by TSD. Upon using the link, applicants create their own users, choose their own password, and set up their own time-based one-time code (TOTP) for 2FA.

TSD admin accounts

TSD’s management project is p01. Staff members at USIT are granted accounts, with the necesarry access rights, according to to following procedure:

• an issue is created in the planning repo: https://github.uio.no/IT-TSD/tsd planning/issues with the name(s) of the people in question

• proseptive users apply via self-service

• prospective users sign the TSD NDA

• the account is approved by TSD staff

• further access is controlled via group memberships

• admin accounts are reviewed during TSD’s quarterly review

TSD user accounts

These are accounts for TSD projects other than the management project. User rights are managed by project admins via group membership.

TSD associated accounts

An associated member is a record of person who has an affiliation with a specific project, but does not have a user account. The person’s record is linked to their 3rd-party login. Their person object can be added to groups to give access to specific users. Persons apply for associated membership to projects using their trusted 3rd-party login, and their applications are managed by project admins.

User deletion

• TSD admins can delete any user, using ur

• Project admins can delete project member user accounts in the self-service portal

Upon deletion, a user’s home directory is archived. If the user is re-created it is unarchived. If not, it is deleted after 90 days.

Handling authentication information

If a TSD staff member must handle authentication information, such as pass words or TOTP secrets they must use approved procedures to communicate the information to the correct person. The following applies:

• credentials can be set using ur, after secure login to a management host in TSD

• if credentials are sent to users they must be encrypted with 7-zip • the decryption key must be communicated via another channel • users should be encouraged to change their credentials after first use

Generic IAM policies

Terms:

• operator - staff member, with privileged access to the IAM • project admin - a user of EC, with admin privileges within a project • project member - a user of EC

• associated member - a person with an affiliation to a project • researcher - a user, with special privileges left unspecified

person objects

• creation

– researchers can create their own person objects, given that they have authenticated with an identity provider with an acceptable level of assurance, such as idporten, or TSD authentication (support for level3 or higher)

– project admins can create persons on behalf of project members, although such persons have no rights until another protocol is used whereby the project member proves that the person belongs to them

– operators can create any person if needed, but in general do not do so as this is a source of error

• granting rights

– persons are granted rights in a project by way of group membership (the iam allows person objects to have group memberships)

– project admins grant project membership (can get a user) 13

– project admins grant associated membership (cannot get a user, but can get other types of project rights - such as that users can share data with them)

• revoking rights

– project admins can revoke access from persons who have an associ ation with their project

– operators can revoke any access from any person

• status management

– operators can expire persons, thereby expiring their user – operators can deactivate persons, thereby deactivating their user – both operations can be reversed

• deletion

– the policy for doing this is not yet decided, needs input from IT-Sec, and legal

user objects

• creation

– project admins can create users in their projects by approving appli cations which have been created as the result of a trusted third-party login

– operators can create any user

• granting rights

– users are granted rights in a project by way of group membership – project admins are moderators of default groups and can add users accordingly

– operators can add any user to any group

• revoking rights

– project admins can revoke group membership if they are moderators – operators can remove any user from any group

• status management

– operators can expire users

– operators can deactivate users

– both operations can be reversed

• deletion

– the policy for doing this is not yet decided, needs input from IT-Sec

group objects

• creation

– operators can create groups

• membership

– project admins can add members based on moderator status – operators can add any member to any group

• moderation

– operators can set moderators

– operators can remove moderators

• status management

– operators can expire groups

– operators can deactivate groups

– both operations can be reversed

• deletion

– operators can delete groups

Project creation

On project creation the following default objects are created:

• pi-group - A group of person PI of the project

• member-group - Group of users members of the project

• admin-group - Group of users administrators of the project • import-group - Group of users which can import files with the file api • export-group - Group of users which can export files with the file api • consent-group - Group of users which can publish files with the consent solution

• publication-group - Group of users which can publish files with the publi cation solution

• people-group - Group of persons which are associated members

Software components

selfservice

Self service web portal

• tsd-selfservice

– tsd-pylib

tsd-cli

Command-line tool for interacting with the Identity and Access Management system.

• tsd-cli

– tsd-pylib

tsd-auth-api

APIs for authentication and authorization.

• tsd-auth-api

– tsd-auth-lib

– pypg-iam

tsd-iam

API for Identity and Access Management,

• tsd-iam

– pypg-iam

– pg-iam

tsd-sentry

2FA API.

• tsd-sentry

tsd-iam-ldap-sync

Synchronise IAM information to Active Directory.

• tsd-iam-ldap-sync

– tsd-pylib

TSD network architecture

All of TSD’s network traffic (ingress and egress) is routed through a set of redundant Cisco switches. A miminal set of ACLs on these switches regulate access to network equipment. Layer 3 firewalling is applied by two sets of redundant FreeBSD routers, one set that governs internal traffic and ipv6 egress, and another set that governs ipv4 ingress and egress.

TSD uses a rfc1918 private address range internally. Layer 2 traffic is segmented into VLANs, and each VLAN has one or more IP subnets addigned to it. Firewall rules applied on the FreeBSD gateway routers regulate inter-VLAN traffic in addition to ingress and egress traffic.

TSD’s project separation is implemented in a multilayerd and hybrid fashion. The administrative project has 11 VLANs with multiple subnets each. Virtual machines which are provisioned in the administrative project run on a sepa rate VMware ESXi cluster. This protects against potential hypervisor attacks launched from project VMs which run on their own VMware ESXi cluster. TSD projects which do not need any inter-VM communication are provisioned in a shared VLAN and are sandboxed from one another via VMware’s NSX-T mi crosegmentation firewall, which is applied on the VM’s virtual network interface. Other projects are provisioned in separate VLANs from one another.

To protect against TCP-over-DNS attacks TSD runs Request Policy Zones (RPZ) on its DNS providers. This limits which DNS names can be resolved inside the TSD network to an approved list.

The Cisco, FreeBSD, and DNS RPZ config files are managed in version control, changes are applied subject to review, and changes are regularly audited at the change council.

Supplier services

Internal Suppliers

Supplier Services Type NETT networking Infrastructure

BSD Data center, Storage, Virtualisation, Virtual

desktop

OPS Operating system management, Active

Directory, NTP, DNS

Infrastructure Infrastructure

DIA Logging, monitoring, Mreg Infrastructure APP OKD, vault Infrastructure BT HPC Infrastructure DBD Databases Infrastructure OKO Procurement, Invoicing Administrative

ABK User, stakeholder, and customer relations

SOJ IT-Sec, UiO-CERT, Legal, Contracts

Administrative Administrative

HJELP Frist line user support Operations UX Web service design Design

Organisational units: https://www.usit.uio.no/om/organisasjon/index.html External suppliers

Supplier Services Type

Digdir ID-porten Identity

SIKT FEIDE Identity

Addressing information security within supplier relation ships

Internal suppliers

The following requirements should be met in these relationships: 17

• USIT suppliers offer services subject to the UiO IT-regulations and Lsis • Any access to TSD that goes beyond normal project membership requires signing the TSD NDA

• Infrastructure providers must:

– document their systems

– get changes approved in the TSD change council

– report production incidents and follow-up items

External suppliers

TSD only uses GDPR compliant external suppliers, and establish Data Processing Agreements where necessary.