Skip to content
BranchesTags

Latest commit

 

History

54 Commits

Folders and files

NameName
Last commit message
Last commit date
README.md
README.md
 
 

Repository files navigation

Contents

SSL/TLS

SSL Testing

This command will display the TLS/SSL protocols that the web server supports:

nmap --script ssl-enum-ciphers -p 443 server.example.com

Example output:

Starting Nmap 7.40 ( https://nmap.org ) at 2020-11-02 19:53 CST
Nmap scan report for server.example.com (192.168.50.50)
Host is up (0.0034s latency).
PORT    STATE SERVICE
443/tcp open  https
| ssl-enum-ciphers:
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 2048) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 2048) - A
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|       Key exchange (dh 2048) of lower strength than certificate key
|       Key exchange (secp256r1) of lower strength than certificate key
|_  least strength: A

Nmap done: 1 IP address (1 host up) scanned in 0.78 seconds

Scanning for Certificate Usage and Expiration By Subnet

This command will scan the specified subnet and look for certificates on the common HTTPS ports 443 and 8443, then filter the output to include the subject, subject alternative name, and expiration timestamp.

This will only find certificates that are configured for the IP or the default server. If there are multiple sites / vhosts at the IP address configured with SNI, they will not be found with this command.

nmap -p 443,8443 -sV -sC 172.21.0.0/24 | grep -E '(Nmap scan report|[0-9]+/tcp|ssl-cert|Subject Alternative Name|Not valid after)'

Validating Certificate Files

Test certificates on a web server using TLSv1.2 and TLSv1.3

curl -vI --tlsv1.2 https://server.example.com
curl -vI --tlsv1.3 https://server.example.com

From https://www.sslshopper.com/article-most-common-openssl-commands.html

# Check a Certificate Signing Request (CSR)
openssl req -text -noout -verify -in CSR.csr
# Check a private key
openssl rsa -in privateKey.key -check
# Check a certificate
openssl x509 -in certificate.crt -text -noout
# Check a PKCS#12 file (.pfx or .p12)
openssl pkcs12 -info -in keyStore.p12

If you use the 'openssl' tool, this is one way to get extract the CA cert for a particular server. This will show the certificate and also evaluate the certificate to show it's details. (using webserver.example.com as an example):

openssl s_client -connect webserver.example.com:443 -servername webserver.example.com </dev/null | openssl x509 -text

The certificate will have "BEGIN CERTIFICATE" and "END CERTIFICATE" markers, and it's details are above the certificate.

If you want to trust the certificate, you can add it to your CA certificate store or use it stand-alone as described. Just remember that the security is no better than the way you obtained the certificate.

Verifying that certificate / private key / certificate signing request all match

(Info taken from https://kb.wisc.edu/page.php?id=4064)

Make sure the output of these 3 commands is the same. If so, then the Certificate / private key / csr match:

openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5
openssl req -noout -modulus -in server.csr | openssl md5

You can also test the cert and key within a pkcs12 file to see if they match the 3 above

openssl pkcs12 -in server.p12 -clcerts -nokeys | openssl x509 -noout -modulus | openssl md5
openssl pkcs12 -in server.p12 -nocerts -nodes | openssl rsa -noout -modulus | openssl md5

Info that expands upon the above:

The private key contains a series of numbers. Two of those numbers form the "public key", the others are part of your "private key". The "public key" bits are also embedded in your Certificate (we get them from your CSR). To check that the public key in your cert matches the public portion of your private key, you need to view the cert and the key and compare the numbers. To view the Certificate and the key run the commands:

openssl x509 -noout -text -in server.crt
openssl rsa -noout -text -in server.key

The 'modulus' and the 'public exponent' portions in the key and the Certificate must match. But since the public exponent is usually 65537 and it's bothering comparing long modulus you can use the following approach:

openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5

Generate a private key

openssl genrsa -out SERVER.key 4096

Generate a CSR

Generating a CSR with a SubjectAlternativeName included in a single line command (This requires that the /etc/ssl/openssl.cnf file exists - Tested on Debian):

openssl req -new -sha256 -key SERVER.key -subj "/C=US/ST=State Name/localityName=City Name/O=Example Inc/emailAddress=youremail@example.com/CN=SERVER.example.com" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:SERVER.example.com,DNS:www.SERVER.example.com,IP:0.0.0.0")) -out SERVER.csr

Links with related info:

https://bugs.chromium.org/p/chromium/issues/detail?id=700595&desc=2 https://bugs.chromium.org/p/chromium/issues/detail?id=308330 https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-command-line https://alexanderzeitler.com/articles/Fixing-Chrome-missing_subjectAltName-selfsigned-cert-openssl/

Generate a CSR without specifying the SubjectAlternativeName attribute (Fine when submitting a request to a public CA)

openssl req -new -sha256 -key SERVER.key -out SERVER.csr
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:State Name
Locality Name (eg, city) []:City name
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example Inc
Organizational Unit Name (eg, section) []:IT Department

Request Certificate from an Active Directory CA

Active Directory CA (Must use IE/Edge Browser): https://subordinate-ca.example.com/certsrv

  1. Request a Certificate
  2. Advanced certificate request
  3. "Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file. "
  4. Paste CSR and choose the appropriate web server certificate template
  5. Make sure to download the Base64 encoded version. Save as .crt

How to create a chained cert file

(order is top down)

  • Server cert (server.example.com)
  • Intermediate Cert
  • Root CA Cert
    • Not necessary…in fact some applications determine it is improper to include the root CA cert in the chain.

Example:

  • server.example.com (Server Certificate)
  • OV_NetworkSolutionsOVServerCA2.crt (Intermediate certificate)

Create PFX/PKCS12 file from PEM cert and key

The .pfx and .p12 file extensions are used interchangeably

Windows Servers tend to want the cert/key files in a pfx/pkcs12 format. Use these commands to create a pfx/pkcs12 file from PEM format key/cert files.

Create PFX/PKCS12 with friendlyname (-name):

openssl pkcs12 -export -out filename.p12 -inkey key-filename.key -in cert-filename.crt -name "friendlyname text"

Create PFX/PKCS12 file with friendlyname (-name) and include cert chain file (-certfile):

openssl pkcs12 -export -out filename.p12 -inkey key-filename.key -in cert-filename.crt -certfile cacert-filename.crt -name "friendlyname text"

Extract Certificate from PFX/PKCS12 file:

openssl pkcs12 -in filename.p12 -nokeys -out cert-filename.crt

Extract Key from PFX/PKCS12 file:

openssl pkcs12 -in filename.p12 -nocerts -nodes -out key-filename.key

Testing Public-Facing Web Servers

HTTPS Testing and Hardening Tools

Trusting a private Root CA in Linux

Debian
Add private Root CA certificate files to /usr/local/share/ca-certificates/, then run this command:

update-ca-certificates

RHEL
Add private Root CA certificate files to /etc/pki/ca-trust/source/anchors/, then run this command:

update-ca-trust extract

Python SSL Trust Issues

If you are experiencing issues with the Python requests module not trusting SSL certificates, ensure the following environment variable is pointed to the correct CA bundle file

# Debian
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

# RedHat
REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt

Example python script to test for SSL Trust issues with the requests module

import requests

def test_https_connection(url='https://google.com'):
    try:
        response = requests.get(url, timeout=5)
        print(f"Status Code: {response.status_code}")
        print("Connection Successful")
    except requests.exceptions.SSLError as e:
        print("SSL Error:", e)
    except Exception as e:
        print("Error:", e)

if __name__ == "__main__":
    test_https_connection()

Web Servers

Apache

Print the current Apache config

apachectl -S

Test the current Apache config for errors

apachectl configtest

List loaded apache modules

apachectl -M

NGINX

Print the current NGINX config

nginx -T

Test the NGINX config file for errors

nginx -t

DNS

Query for Active Directory Domain Controller SRV records

Example using dig to find SRV records for the example.com domain:

    user@workstation:~$ dig +noall +answer srv _ldap._tcp.dc._msdcs.example.com
    _ldap._tcp.dc._msdcs.example.com. 536   IN SRV  0 100 389 dc1.example.com.
    _ldap._tcp.dc._msdcs.example.com. 536   IN SRV  0 100 389 dc2.example.com.

Clearing DNS records from Active Directory cache

This is useful for DNS environments where Active Directory DNS is configured to perform lookups to another DNS system that is authoritative for internal DNS records. This will describe clearing the cache for individual records rather than the entire DNS cache

Run from an AD DNS Server

  1. Log into the AD DNS server that you want to clear the cached record from
  2. Run the following commands in Powershell depending on the type of record you are working with

    • Repeat for each AD DNS server that holds the cached record

      A Record

      Query for the record to see if it exists:

            Get-DnsServerResourceRecord -ZoneName ..cache -RRType A -Name server.example.com

      Remove the record:

            Remove-DnsServerResourceRecord -ZoneName ..cache -RRType A -Name server.example.com

      CNAME Record

      Query for the record to see if it exists:

            Get-DnsServerResourceRecord -ZoneName ..cache -RRType CNAME -Name server-cname.example.com

      Remove the record:

            Remove-DnsServerResourceRecord -ZoneName ..cache -RRType CNAME -Name server-cname.example.com

Run Remotely from a Workstation (Windows 8 or above REQUIRED)

Alternatively, these commands can be run from a regular workstation as long as Powershell is launched with a user account that has permission to modify the records. The following switch must be appended to the commands, and run once for each AD DNS server (dc1.example.com, dc2.example.com)

    -ComputerName dc1.example.com

Remote Server Administration Tools may need to be installed so that the required module is available to Powershell on your workstation. https://www.microsoft.com/en-us/download/details.aspx?id=45520

You can see if you have the necessary module by running this command in Powershell.

    Get-Module -ListAvailable DNSServer

Info gathered from:

Git

List Remotes

git remote -v

Clean changes from repo including files that match .gitignore

git clean -Xdf

Add execute permissions to a file in a git repository

# Show permissions of files in the current directory of the repository
git ls-files --stage

# Add the execute permission to script.sh
git update-index --chmod=+x script.sh

# Add commit message and push
git commit -m "Add execute permission to script.sh"
git push

Debugging git CLI comamnds

Enable trace logging with git commands

  • In Linux (bash): `GET_TRACE=1
  • In Windows (CMD): set GIT_TRACE=1
  • In Windows (Powershell): `$env:GIT_TRACE=1

Increase the verbosity of the git SSH command

GIT_SSH_COMMAND="ssh -vvv" git fetch

The loglevel can be turned up on per host in ~/.ssh/config:

Host github.com
        LogLevel DEBUG3

Show all current configuration

git config --global --list

Gitlab Specific

Generate list of active users and email addresses from Gitlab API

I used this API query once in a while to grab the e-mail addresses of all active users on a Gitlab instance.

curl -L --header "PRIVATE-TOKEN: <REPLACE WITH VALID TOKEN>" "https://gitlab.example.com/api/v4/users?active=true&per_page=500" | jq -r '.[] | .email' | sort

Documentation: https://docs.gitlab.com/ee/api/users.html

Purging data from Git repositories

If large binary files or executables have been stored in a git repository and you would like to clean them, this tutorial from Gitlab is helpful:
https://docs.gitlab.com/17.0/ee/user/project/repository/reducing_the_repo_size_using_git.html

This requires the git-filter-repo package which is only available in Debian 10+ and Ubuntu 22+. Alternatively, it can be downloaded directly from the source repository and ran with python3 git-filter-repo

This is the actual command I used to remove the large files from the repo (in step 8 of the Gitlab doc)

git filter-repo --invert-paths --path path/to/folder --path path/to/file1 --path path/to/file2

After running the "git filter-repo" command, there is a file in the project directory under "filiter-repo/commit-map". This file needs to be preserved and uploaded to the Gitlab server during the "Repository cleanup" section at the bottom of the document. Without this, the Gitlab server won't actually reduce the size of the repo.

After running through this process, I was able to drop the repo size from 700MB to 5MB.

NOTE: This process can also be used to delete files containing sensitive information from repositories, but it MAY not completely remove it.

SaltStack

Useful Commands

Watch the Salt Event Bus:

salt-run state.event pretty=True

Refresh fileserver immediately:

salt-run fileserver.update

View directory list for an environment:

salt-run fileserver.dir_list saltenv=test

View file list for an environment:

salt-run fileserver.file_list saltenv=test

Troubleshoot a highstate run:

salt-call -l debug state.apply

Look up recent job IDs and view the output of one:

salt-run jobs.list_jobs
salt-run jobs.lookup_jid <job id number>

Show currently running jobs:

salt-run jobs.active

MySQL

MySQL Common Commands

Creating a database and assigning a user all privileges:

    CREATE DATABASE exampledb_dev;
    CREATE USER 'exampleuser'@'%' IDENTIFIED BY '$password';        # % means the user can log in from any location
    GRANT ALL PRIVILEGES ON exampledb_dev.* TO 'exampleuser'@'%';
    FLUSH PRIVILEGES;

Checking grants:

    show grants for 'exampledb'@'%';
    FLUSH PRIVILEGES;

Revoke a single privilege from a single database (Database name must be specified): (note the use of backticks around the database name)

    REVOKE CREATE VIEW on `exampledb\_dev`.* FROM 'exampledb_dev'@'%';

Changing host for a user:

    UPDATE mysql.user SET host = '%' WHERE user = 'exampleuser';

Show all users:

    select user,host from mysql.user;

Drop a database and user:

    DROP DATABASE exampledb_dev;
    DROP USER 'exampleuser'@'%';

Show variables for innodb engine;

    SHOW VARIABLES LIKE 'innodb_file_format%';

Set default character set and collation of a database:

    ALTER DATABASE dbname DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

Look at views that exist on all databases:

    select * FROM information_schema.views;

Remove a single view from a database:

    drop view exampledb_dev.exampleview;

Determine the amount of disk space being used by each database on a server:

    SELECT table_schema AS "Database", SUM(data_length + index_length) / 1024 / 1024 AS "Size (MB)" FROM information_schema.TABLES GROUP BY table_schema;

Show all stored procedures on a database server:

    SHOW PROCEDURE STATUS;

Show current connections to a database:

    SHOW PROCESSLIST;

Kill off query that is being problematic:

    Determine the "ID" of the query that needs to be killed
            SHOW PROCESSLIST;

    Kill the offending query:
            kill <query ID>;

Repair a crashed table:

    repair table table_name;

Extracting a table from a mysqldump file

Extract the table ‘exampleTable’ from the gzipped database dump file exampledb_12232021.sql (generated from the 'exampledb' database)

    zcat /var/backups/mysql/exampledb_20211223.sql.gz | sed -n -e '/CREATE TABLE.*`exampleTable`/,/CREATE TABLE/p' > exampleTable_12232021.sql

Delete the drop/create lines of the next database from the bottom of the file (manually)

vim exampleTable_12232021.sql

Rename the table to something else so it can be re-imported:

    sed -i 's/`exampleTable`/`exampleTable_12232021`/g' exampleTable_12232021.sql

Re-import the table to the database with the new name:

mysql -uroot -p"$(</root/.sql_passwd)" exampledb < exampledb_12232021.sql
  • Might need to manually copy the lines from just above the 'CREATE TABLE' line which configure the character set, etc.

SMTP

testing smtp with STARTTLS

# Create a file with these contents named `testmessage`
# Substitute your own from and to addresses
EHLO example.com
MAIL FROM: noreply@example.com
RCPT TO:testuser1@example.com
RCPT TO:testuser2@example.com
DATA
From: noreply@example.com
To: testuser1@example.com,testuser2@example.com
Subjet: Test Email
The body goes here
.
QUIT

# Use this openssl command to send the contents to the destination using STARTTLS
openssl s_client -starttls smtp -connect smtp.example.com:597 -crlf < testmessage

add another example for testing plain-text SMTP with netcat or telnet

Networking

Managing TCP Sessions

Managing TCP Connections To kill a currently established TCP connection, the following command can be used: (using a destination ip/port of 192.168.50.50 and 389 as an example

    ss -K dst 192.168.50.50 dport = 389

Clearing ARP cache

ESXi 5.1

View ARP cache:

    esxcli network ip neighbor list

Remove an item from ARP cache:

    vsish -e set /net/tcpip/v4/neighbor del IPADDRESS

Debian

View ARP cache:

    arp

View ARP cache (IP only, don't resolve hostnames):

    arp -n

Remove an item from ARP cache:

    arp -d IPADDRESS

Gratuitous ARP

Useful when replacing a server and the new one has a new MAC address. This will announce the new MAC address that is associated with the IP address to devices on the local network:

    arping -A -I eth0 IPADDRESS

View Network Interface Details (Linux)

Using ethtool to view the "ens18" interface:

[root@server ~]# ethtool ens18
Settings for ens18:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Supported pause frame use: No
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full
							100baseT/Half 100baseT/Full
							1000baseT/Full
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: off (auto)
	Supports Wake-on: umbg
	Wake-on: d
		Current message level: 0x00000007 (7)
							drv probe link
	Link detected: yes

Monitoring live bandwidth usage on a specific interface

iftop -i bond0

VMware PowerCLI

Gather VM names that have snapshots from VMware with a powershell module (requires vmware powercli) connect-viserver vcenter.example.com

    get-vm | get-snapshot | format-list vm, name, description, created, sizegb | out-file snapshots.csv

Linux - Useful Commands

Debian

List files within a Deb package

  • Downloaded Deb Package:
    dpkg --contents <rpmname>.deb
  • Installed Deb Package:
    dpkg -L <package name>

Determine the package a file comes from:

    dpkg -S /path/to/file

Find Package Dependencies

  • Recursive dependencies for a package:
    apt depends --recurse <package name>
  • Reverse dependency lookup based on installed packages:
    apt rdepends --installed

Package Cleanup

Remove orphaned packages

    apt autoremove --purge

Remove config files for packages that were removed but not purged

Confirm they are no longer needed before proceeding with removal!

    apt-get purge $(dpkg -l | grep -E '^rc' | awk '{print $2}' | while read line; do printf "$line "; done)

Post-Upgrade Cleanup Command

After running OS versions, use this command to purge packages that were build for previous versions of Debian. Confirm
these packages are longer needed and/or determine an upgrade path for the packages before removal!

    apt-get purge $(dpkg -l | grep -E '(deb9|stretch|deb10|buster|deb11|bullseye|^rc)' | awk '{print $2}' | while read line; do printf "$line "; done)

Remove downloaded packages to clean up disk space

    apt clean

RHEL

View file/folder permissions set by package (NGINX as an example)

    rpm -q --queryformat="[%{FILEMODES:perms} %{FILENAMES}\n]" nginx

Clean up old kernels (Preserving the current and 1 previous version):
https://access.redhat.com/solutions/1227

  • RHEL 5/6/7 (requires yum-utils package):

    package-cleanup --oldkernels --count=2

  • RHEL 8/9:

    dnf remove $(dnf repoquery --installonly --latest-limit=-2 -q)

List enabled/disabled/all repositories

    yum repolist enabled
    yum repolist disabled
    yum repolist all

Disable a repository for a single yum transaction Useful if the repository is being problematic

    yum --disablerepo="reponame" info openssh

    Alternatively, a repo can be enabled for a single yum transaction by using the --enablerepo option

Show all available versions of a package

    yum --showduplicates list <package name>

Install specific version of a package

    yum install <package name>-<version number>

List files within an RPM package

  • Downloaded RPM Package:
    rpm -qlp <name>.rpm
  • Installed RPM Package:
    rpm -ql <package name>
  • Package in a configured repository that hasn't been installed:
    dnf repoquery --list <package name>

Determine the package a file comes from:

  • yum whatprovides /path/to/file
  • rpm -qf /path/to/file
  • dnf provides /bin/ps

List all packages that are installed from a specific repo (using the EPEL repo as an example):

  • dnf list installed | grep @epel
  • dnf repo-pkgs epel list installed

Finding Package Dependencies

  • With an rpm file:
    rpm -qpR <package name>.rpm
  • With an installed package:
    rpm -qR <package name>
  • With repoquery (included in the dnf-utils or yum-utils package):
    repoquery --requires --resolve <package name>
  • With repoquery (recursive):
    repoquery --requires --resolve --recursive <package name>
  • With repoquery (reverse lookup):
    repoquery --whatdepends <package name> --installed

List what capabilities a package provides

  • rpm -q --provides <package name>

Replace one similar/equivalent package with another

  • dnf --allowerasing <new package>
  • dnf swap <old package> <new package>

List all packages available in all enabled repositories:

    dnf list --all

View Package Changelog (from repo):

    dnf changelog <package name>

View Package Changelog (currently installed package):

    rpm -q --changelog <package name>

Simulate Updates:

    dnf update --assumeno

Install only package updates that resolve a CVE or multiple CVEs:

    dnf update --cve=CVE-####-####
    dnf update --cves=CVE-####-####,CVE-####-####,CVE-####-####

Install only package updates that resolve an advisory:

    dnf update --advisory=RHSA-XXXX:XXXX
    dnf update --advisories=RHSA-XXXX:XXXX,RHSA-XXXX:XXXX,RHSA-XXXX:XXXX

Install all security updates updates:

    dnf update --security

List security updates that have been installed on a server:

    dnf updateinfo security --installed

Package Cleanup

Display a list of orphaned packages (requires the dnf-utils package)

    package-cleanup --leaves

Cleaning up duplicate pacakges

# List duplicate packages installed on the server:
dnf repoquery --duplicates

# These can be cleaned up with:
package-cleanup --cleandupes (requires yum-utils package)

Remove orphaned packages

    dnf autoremove

Remove downloaded packages to clean up disk space

    dnf clean packages

Other Useful RHEL Links:

RPM Database Corruption

Fix yum/dnf if the command "hangs" and does not return output, or you receive errors that the RPM database is corrupted

Check for processes holding the RPM database open. Kill any processes that are listed
	lsof | grep /var/lib/rpm
Make a backup of the RPM database before making changes
	cp -a /var/lib/rpm /var/lib/rpm.bak
Delete the RPM db lock files
	rm -f /var/lib/rpm/__db*
Rebuild the RPM indexes
	rpm -vv --rebuilddb
Verify the RPM database
	cd /var/lib/rpm
	/usr/lib/rpm/rpmdb_verify Packages
Clean dnf cache
	dnf clean all
		
More Info Here
https://access.redhat.com/solutions/6903

SSH

Bash configuration for configuring your SSH agent for ~/.ssh/id_rsa and proxying connections to servers through a basion host / jump box:
The jumpbox must be configured to allow ssh agent forwarding

Add to ~/.bashrc

# Set up SSH Auth Agent and Key
SSH_AUTH_SOCK="$HOME/.ssh/ssh-agent.sock"
export SSH_AUTH_SOCK

if ! pgrep -u "$USER" ssh-agent > /dev/null; then
    rm -f "$SSH_AUTH_SOCK"
    ssh-agent -a "$SSH_AUTH_SOCK" > /dev/null
fi

if ! ssh-add -l | grep '.ssh/id_rsa' > /dev/null; then
    ssh-add ~/.ssh/id_rsa
fi

Add to ~/.ssh/config

Host jumpbox
    Hostname jumpbox
    User username
    ForwardAgent yes

# Adjust server???? to match your naming convention
Host server???? !jumpbox
    Hostname %h
    User username
    ProxyJump jumpbox

Systemd

Viewing the memory usage of a unit

  • MemoryCurrent - Current memory usage
  • MemoryPeak - Peak memory usage (not always available)
  • MemoryMax - Memory limit (infinity if there is no limit)
systemctl show <unit name> -p MemoryCurrent -p MemoryPeak -p MemoryMax

Linux Benchmarking

The tools below can be found in Debian and RHEL repositories

CPU and RAM Benchmarking Tool: sysbench

CPU Test

    sysbench cpu --threads=<number of cores> run

RAM Test

    sysbench memory run

Storage Benchmarking Tool: fio

Sequential Reads

    sync; fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=/path/to/testfile --bs=4k --size=4G --readwrite=read --ramp_time=4

Sequential Writes

    sync; fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=/path/to/testfile --bs=4k --size=4G --readwrite=write --ramp_time=4

Random Reads

    sync; fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=/path/to/testfile --bs=4k --size=4G --readwrite=randread --ramp_time=4

Random Writes

    sync; fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=/path/to/testfile --bs=4k --size=4G --readwrite=randwrite --ramp_time=4

Random Reads and Writes

    sync; fio --randrepeat=1 --ioengine=libaio --direct=1 --name=test --filename=/path/to/testfile --bs=4k --size=4G --readwrite=randrw --rwmixread=70 --ramp_time=4

Network Benchmarking Tool: iperf3

You will need two servers for this test. One will act as the client and one will act as the server. These commands will run a 30 second test showing the speed that can be achieved between the two systems.

Server Command

    iperf3 -s -p 5201

Client Command

    iperf3 -c <ip of server running iperf3> -p 5201 -t 30s

    # Run with more streams to help saturate bandwidth (4 streams)
    iperf -c <ip of server running iperf3> -p 5201 -t 30s -P 4

Linux Memory

Memory testing on a machine where you can't run memtest86 (like a router where you have shell access)

# Install 'memtester' package
# Check available RAM first
free -h
# Run test (leave ~1GB headroom for the OS)
# syntax: memtester <size> <passes>
memtester 8G 1

Linux Storage

Finding Disk Usage

TUI Utility similar to WinDirStat on Windows

    ncdu

Get summary of disk usage of top level directories under / while avoiding paths that will just give undesirable output

    du -hs --exclude=/dev --exclude=/proc --exclude=/run --exclude=/sys /*

Check home directories to see what large files were created in the past day:

    find /home -size +100M -mtime -1 -exec du -hs {} \;

Disk is filling up, but running 'du -hs' on the directory doesn't show what is using the disk space.

In one case I found that rsyslogd was holding files open that were supposed to be deleted and were filling up the /var partition. To resolve the issue, I just had to restart rsyslog so it would let go of the file handles and allow them to be deleted. I used the following command to find that out:

    lsof | grep "/var" | grep deleted

If files cannot be written and df -h shows free space, check inode utilization

    df -i

Rescan the SCSI Bus

This command will poke at the SCSI controllers to look for changes. Helpful for detecting resized virtual disks without rebooting the VM.

    for i in /sys/class/scsi_device/*/device/rescan; do echo '- - -' >"$i"; done

The sg3_utils package in RHEL-based distributions provide a command that performs the same function

    scsi-rescan

Rescan Fibre Channel Connections (Loop Initialization Protocol)

    for host in /sys/class/fc_host/host*; do echo "Rescanning $host"; echo "1" > "$host/issue_lip"; done

Securely wiping a disk with Shred

The shred command can be used to securely wipe a disk. This command should be available on most systems. This command will make 3 passes of writing random data to the device, then a single pass of writing 0's to the device to hide the fact that it has been wiped.

    shred -vfz /dev/(device name without partition number)

Showing storage device details

List attached disks and their details

    lshw -class disk

List block devices

    lsblk

List block devices with filesystem information

    lsblk -f

View current multipath devices

    multipath -ll

Reload multipath device mappings

    multipath -r

Flush all unused multipath device mappings

    multipath -F

Smartctl commands

Check health of a single device

    smartctl -H /dev/sda

Show all devices found by smartctl

    smartctl --scan
    
    # Example Output
    /dev/bus/0 -d megaraid,1 # /dev/bus/0 [megaraid_disk_01], SCSI device
    /dev/bus/0 -d megaraid,2 # /dev/bus/0 [megaraid_disk_02], SCSI device
    /dev/bus/0 -d megaraid,3 # /dev/bus/0 [megaraid_disk_03], SCSI device
    /dev/bus/0 -d megaraid,4 # /dev/bus/0 [megaraid_disk_04], SCSI device

Show data for a single device accessed through a MegaRAID controller

    smartctl -a -d megaraid,4 /dev/bus/0

Using storcli to show all info for controller 0 (MegaRAID Controllers)

    storcli /c0 show all

ZFS

Monitor IO Statistics on a ZFS Zpool (read/write IOPS and bandwidth)

zpool iostat -v 2

Monitoring Processes for Storage Issues

Using the top command to view stalling processes:

  • Open the top command
  • Type o and add the filter S=D, then hit enter.
  • Show full process string - Type c
  • Processes showing the D state are getting stuck waiting on I/O

Viewing process io statistics:

cat /proc/\<pid>/io

View which files a process has open

ls -l /proc/<pid>/fd/

Monitor local storage devices for performance issues (from sysstat package)

watch "iostat -xz"
# Look for
#   - High %util
#   - Large await times
#   - Growing queue depths

Show process-level IO stats (samples every second)

pidstat -d 1 -p <pid>

Linux Miscellaneous

Find command that avoids multiple paths

find / \( -path "/mount1" -o -path "/mount2" -o -path "/mount3" \) -prune -o -iname 'filename.yaml'

Run an strace on a running process to see what it is doing

strace -p <pid>
# Multiple processes
strace -p <pid> -p <pid> -p <pid>
# Follow forks and child processes
strace -fp <main pid>

Determining the purpose of a server

I usually use a combination of looking at running services in systemctl list-units, which ports have processes bound to them or sockets established with netstat -nlp, look at crons that are configured with crontab -l or looking at what exists in /var/spool/cron and then follow breadcrumbs from there.

Commands to determine if a server is physical or virtual (by order of likeliness to exist on the system)

    dmidecode -s system-manufacturer
    systemd-detect-virt
    virt-what

Out of Band Management

IPMI

ipmitool commands

Can be run on the same system as the OOBM is located on

# View the system event log
ipmitool sel list
# Clear the system event log
ipmitool sel clear
# View hardware sensor readings
ipmitool sensor list
# Perform a cold reset of the BMC
ipmitool mc reset cold

Other options are needed if you are running the commands on a separate system that has access to the OOBM IP

ipmitool -I lanplus -H <OOBM_IP> -U <USER> -P '<PASS>' sel list

Containers

Pulling a container image from one registry and pushing it to another

(Assuming authentication is already in place)

# Pull container image from the source registry
docker pull ghcr.io/clayoster/dnsquery:latest
# Tag image with the destination registry path
docker tag ghcr.io/clayoster/dnsquery:latest hub.docker.com/clayoster/dnsquery:latest
# Push to the destination registry
docker push hub.docker.com/clayoster/dnsquery:latest

Docker image build commands:

# Force an amd64 image to be pulled to a arm64 machine (like an M series Macbook)
docker pull ghcr.io/clayoster/dnsquery:latest --platform linux/amd64

# Initiate an image build
docker buildx build --target prod -t dnsquery:devtest .

# Run that image 
docker run --rm -p 127.0.0.1:8080:8080 dnsquery:devtest

Container Scanning with Trivy

# Trivy scan an image for critical and high vulnerabilities
trivy image -s=CRITICAL,HIGH dnsquery:latest

# Want to force trivy to use an image that is built locally? Specify the hash/id from docker image ls
trivy image -s=CRITICAL,HIGH d939231bc670

Container Hardening

Use the examples below to add container hardening. Beware these settings will probably break things.

Docker Compose - Add these settings

# Hardening settings
read_only: true
privileged: false
cap_drop:
  - ALL
# If necessary, add capabilities back in
#cap_add:
#  - See https://man7.org/linux/man-pages/man7/capabilities.7.html) for options
security_opt:
  - 'no-new-privileges=true'
# Necessary if the app needs to write to /tmp
#tmpfs:
#  - /tmp

Kubernetes Pod Manifests - Add these settings

spec:
   containers:
     - name: example
       image: alpine:latest
       securityContext:
         allowPrivilegeEscalation: false
         capabilities:
           drop:
             - ALL
           # If necessary, add capabilities back in
           #add:
           #  - See https://man7.org/linux/man-pages/man7/capabilities.7.html) for options
         privileged: false
         readOnlyRootFilesystem: true
         runAsNonRoot: true
         seccompProfile:
           type: RuntimeDefault
   # Necessary if the app needs to write to /tmp
   #    volumeMounts:
   #      - name: tmp
   #        mountPath: /tmp
   #volumes:
   #  - name: tmp
   #    emptyDir:
   #      medium: Memory

Kubernetes

Kubectl

These commands assume that a kubeconfig has been set (ex. export KUBECONFIG=/path/to/your/kubeconfig)

View all nodes in the cluster

kubectl get nodes

View all nodes in a cluster with their CPU and Memory capacities

kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory

View cluster component statuses (scheduler, controller-manager, etcd)

kubectl get componentstatus

or

kubectl get --raw='/readyz?verbose'

View all pods in the cluster

kubectl get pods -A

# All pods except the kube-system namespace
kubectl get pods --all-namespaces --field-selector metadata.namespace!=kube-system

# With more detail
kubectl get pods -A -o wide

# Watch for changes
kubectl get pods -A -o wide -w

View all services in the cluster

kubectl get svc -A

View all pods, services, and ingresses in the cluster

kubectl get pods,svc,ingress -A

List all api resources (kinds) with verbs

kubectl api-resources -o wide

Follow the logs for a pod

kubectl logs -f -n <namespace name> <pod name>

Follow the logs based on an application label

kubectl logs -f -n <namespace name> -l app=<app name>

Edit a config map from the cli

# Find the configmap
kubectl get configmaps -A

# Edit the configmap
kubectl edit configmap -n <namespace name> <configmap name>

Apply a Kubernetes manifest

kubectl apply -f manifest.yaml

Restart a deployment

kubectl rollout restart deployment -n <namespace name> <deployment name>

Map local port to a port within a pod for direct access
local port 8080, pod port 80

kubectl port-forward nginx-pod-0 8080:80

Misc

Create a Secret for imagePullSecrets

Generate a secret for authentication to a container registry. It will print a secret manifest that can be configured in the appropriate namespace and used for imagePullSecrets for pulling container images.

kubectl create secret docker-registry registry-auth-secret-name \
  --docker-server=git.example.com \
  --docker-username=insertusernamehere \
  --docker-password=insertpasswordhere \
  --namespace=yournamespacehere \
  --dry-run=client -o yaml

View all pods in a namespace with their container images

kubectl get pods -n <namespace> -o custom-columns='POD:.metadata.name,NODE:.spec.nodeName,IMAGE:.spec.containers[*].image,IMAGE_ID:.status.containerStatuses[*].imageID'

Print a CSV list of namespaces and the container images running within them

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}' | sort | uniq

Determine which pod that a filesystem path beneath like /var/lib/kubelet/pods/\<pod-uid>/path/to/file relates to

kubectl get pods -n <namespace> -o custom-columns=NAME:.metadata.name,UID:.metadata.uid | grep <pod-uid>

Troubleshooting

Launch a shell into running pod (assuming /bin/sh is available)

kubectl exec -it -n <namespace name> <pod name> -- /bin/sh

Manually run a container in 'my-namespace' for debugging
Using Netshoot as the container image

kubectl run -it debugging-pod --image=nicolaka/netshoot --restart=Never -n default -- sh

# Delete the pod after you are finished
kubectl delete pod debugging-pod -n my-namespace

Find all pods that are not owned by a controller (These cause issues with draining nodes)

kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | [.kind, .metadata.namespace, .metadata.name, .spec.nodeName] | @tsv'

Run in a loop to delete the pods without controllers across all cluster nodes

# Find and delete pods without controllers defined
while read kind namespace name node; do
    kubectl delete pod "$name" -n "$namespace" --force --grace-period 1
done < <$(kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | [.kind, .metadata.namespace, .metadata.name, .spec.nodeName] | @tsv')

View the events occurring across the cluster

kubectl get events -A

Testing RBAC that is applying to a service account

kubectl auth can-i delete replicaset/testing -n testing --as=system:serviceaccount:test-service-account

Helm

Show the versions of a chart in a helm repo

helm search repo helm-repo-name/chart-name --versions

Show the values that were supplied with an existing helm deployment

helm get values helm-deployment-name -n namespace

# Output the values to a file
helm get values rancher -n cattle-system -o yaml > values.yaml

Output the manifests that are currently managed by a helm deployment

helm get manifest helm-deployment-name -n namespace

Apply a helm chart using a values file

helm upgrade helm-deployment-name helm-repo/helm-chart -n namespace --version=X.X.X -f values.yaml

Download and unpack a helm chart, edit something and apply the modified version

helm fetch helm-repo/helm-chart --version=X.X.X --untar
# edit the file(s) you need to inside the helm-chart directory
helm upgrade helm-deployment-name ./helm-chart --version=X.X.X -n namespace

Output the CRDs from a specific helm chart version

    helm repo update
    helm search repo helm-repo-name/chart-name --versions
    helm show crds helm-repo-name/chart-name --version x.x.x

Helm Chart Creation Commands

# Scaffold helm project called "helm-test"
helm create helm-test

cd helm-test

# Edit files within the chart directory
# https://helm.sh/docs/topics/charts/#the-chart-file-structure

# Test that the chart is well formed
helm lint

# Locally render the template
helm template my-release .

# Enable verbose output
helm template my-release . --debug

# Package the chart
# The chart and app version numbers come from the Chart.yml file at the root of the chart directory
cd ..
helm package helm-test/

Cilium

Check cilium status from the cilium-cli tool (assuming it is installed)

cilium status

Find the cilium pod names

kubectl -n kube-system get pods -l k8s-app=cilium

View cilium status

kubectl exec -it -n kube-system <cilium pod name> -- cilium-dbg status

Check hubble status

kubectl exec -it -n kube-system <cilium pod name> -- cilium-dbg status

Follow all traffic flows with hubble

kubectl exec -it -n kube-system <cilium pod name> -- hubble observe -f

Follow traffic flows for a specifc pod with hubble

kubectl exec -it -n kube-system <cilium pod name> -- hubble observe -f --pod <namespace name>/<pod name>

Using the cilium-cli and hubble commands (installed via brew or other)
Hubble Documentation

# Forward hubble to a port on your local system so you can run hubble commands
cilium hubble port-forward

# View traffic flows
hubble observe -f

# Watch traffic in specific namespaces on a multiple ports
hubble observe -f -n namespace1 -n namespace2 --port 80 --port 443

# Watch for dropped traffic in a cluster (Indication of network policy not allowing the connection)
hubble observe -f -n namespace1 --port 443 --type drop

Launch the Hubble Web UI - Visualizes traffic flows (filters for namespace and action)
Automatically launches the page in a web browser

cilium hubble ui

Kustomize

Test building kustomize paths:

kubectl kustomize path/to/kustomization

Only show a manifest with a specific name from the kustomization:

kubectl kustomize path/to/kustomization | yq eval 'select(.metadata.name == "name-of-manifest")'

Show multiple manifests with different names from the kustomization:

kubectl kustomize path/to/kustomization | yq eval 'select(.metadata.name == "name-of-manifest" or .metadata.name == "name-of-second-manifest")'

Only show kustomizations of the "kind" of Namespace:

kubectl kustomize path/to/kustomization | yq eval 'select(.kind == "Namespace")'

Sort YAML documents by .metadata.name using yq

kubectl kustomize path/to/kustomization | yq eval '(.kind = "Namespace")' | yq eval-all '. as $item ireduce ([]; . + [$item]) | sort_by(.metadata.name)[]'

Flux

List all flux-managed resources across all namespaces

flux get all -A

Reconcile a specific kustomization

flux reconcile kustomization <kustomization-name>

Follow the flux controller logs

kubectl logs -f -n flux-system deploy/notification-controller

WSL - Windows Subsystem for Linux

Disable Terminal Beep

Disable the source of the beeps

from https://stackoverflow.com/a/36726662/5145596

Note: These only apply to the local terminal and will not apply to SSH sessions

  1. To disable the beep in bash you need to uncomment (or add if not already there) the line set bell-style none in your /etc/inputrc file.

  2. To disable the beep and the visual bell also in vim you need to add the following to your ~/.vimrc file:

     set visualbell
 set t_vb=

  3. To disable the beep also in less (i.e. also in man pages and when using "git diff") you need to add export LESS="$LESS -R -Q" in your ~/.profile file.

Completely Disable the terminal beep sound in Windows Terminal

This will apply to SSH sessions
In Windows Terminal, go to Settings > Profiles > Defaults > Additional Settings > Advanced > Bell Notification Style and unset "audible" here. The corresponding setting in the settings.json file for Windows Terminal look like this:

{
  "profiles": {
    "defaults": {
      "bellStyle": "none"
    }
  }
}

bellStyle may also include "window" or "taskbar" which may be acceptable as they do not trigger sounds.

Disable the terminal beep sound completely in Windows

The "Critical Stop" sound in Windows is what is played when terminal beeps occur. Setting the sound to "(none)" doesn't disable the sound, but instead causes a different default sound to be played. A better option is to generate a slient WAV format file and set that as the sound for "Critical Stop"

  1. Install the sox package (Swiss army knife of sound processing)
  2. Run this command to generate a file named "silence.wav" that contains .5 seconds of silence: sox -n -r 44100 -c 2 slience.wav trim 0.0 0.5
  3. Move the file to a location where Windows can access it
  4. Open "Change system sounds" from the control panel, locate the "Critical Stop" sound and set it to the silence.wav file that was created. Click Apply

About

A collection of useful commands

Resources

Readme
Activity

Stars

2 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors