DevOps-Bash-tools

Hari Sekhon - DevOps Bash Tools

git.io/bash-tools

1200+ DevOps Shell Scripts and Advanced Bash environment.

Fast, Advanced Systems Engineering, Automation, APIs, shorter CLIs, etc.

Heavily used in many GitHub repos, dozens of DockerHub builds (Dockerfiles) and 600+ CI builds.

Summary

• Scripts for many popular DevOps technologies, see Index below for more details
• Advanced configs for common tools like Git, vim, screen, tmux, PostgreSQL psql etc...
• CI configs for most major Continuous Integration products (see CI builds page)
• CI scripts for a drop-in framework of standard checks to run in all CI builds, CI detection, accounting for installation differences across CI environments, root vs user, virtualenvs etc.
• API scripts auto-handling authentication, tokens and other details to quickly query popular APIs with a few keystrokes just supplying the /path/endpoint
• Advanced Bash environment - .bashrc + .bash.d/*.sh - aliases, functions, colouring, dynamic Git & shell behaviour enhancements, automatic pathing for installations and major languages like Python, Perl, Ruby, NodeJS, Golang across Linux distributions and Mac. See .bash.d/README.md
• Installs the best systems packages - AWS CLI, Azure CLI, GCloud SDK, Digital Ocean CLI, Terraform, Terragrunt, GitHub CLI, Kubernetes kubectl & kustomize, Helm, eksctl, Docker-Compose, jq and many others... extensive package lists for servers and desktops for most major Linux distributions package managers and Mac
  • install/ - contains many installation scripts for popular open source software and direct binary downloads from GitHub releases
  • configs/ - contains many dot configs for common technologies like ViM, top, Screen, Tmux, MySQL, PostgreSQL etc.
  • setup/ - contains setup scripts, package lists, extra configs, Mac OS X settings etc.
• Utility Libraries used by many hundreds of scripts and builds across repos:
  • .bash.d/ - interactive library
  • lib/ - scripting and CI library
• SQL Scripts - 100+ scripts for PostgreSQL, MySQL, AWS Athena + CloudTrail, Google BigQuery
• Templates - templates for common programming languages and build configs
• Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

See Also: similar DevOps repos in other languages

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

(you're welcome to connect with me on LinkedIn)

Quick Setup

To bootstrap, install packages and link in to your shell profile to inherit all configs, do:

curl -L https://git.io/bash-bootstrap | sh

• Adds sourcing to .bashrc/.bash_profile to automatically inherit all .bash.d/*.sh environment enhancements for all technologies (see Inventory below)
• Symlinks .* config dotfiles to $HOME for git, vim, top, htop, screen, tmux, editorconfig, Ansible, PostgreSQL .psqlrc etc. (only when they don't already exist so there is no conflict with your own configs)
• Installs OS package dependencies for all scripts (detects the OS and installs the right RPMs, Debs, Apk or Mac HomeBrew packages)
• Installs Python packages
• Installs AWS CLI

To only install package dependencies to run scripts, simply cd to the git clone directory and run make:

git clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools
cd bash-tools
make

make install sets your shell profile to source this repo. See Individual Setup Parts below for more install/uninstall options.

Index

• Dot Configs - .gitconfig, .vimrc, .screenrc, .tmux.conf, .toprc, .gitignore...
• Bash Environment & Libraries - .bashrc, .bash.d/ interactive library, lib/ scripting library
• Installation Scripts for many popular open source technologies
• Linux & Mac - curl OAuth / JWT, LDAP, find duplicate files, SSL certificate get/validate, URL encoding/decoding, Vagrant
• Mac & AppleScript - Mac settings and UI automation scripts, send keystrokes, mouse clicks, detect foreground app, switch app, detect locked screen or screensaver, activate screensaver, Hammerspoon system event handlers such as automatically switching audio to be able to Shazam while watching on AirPods
• Monitoring - Grafana, Prometheus, Node Exporter, scripted collection of common Linux & Mac cli monitoring stats and log locations for quick generation of vendor support tarball bundles both locally and over SSH
• AWS - Amazon Web Services - AWS account summary, lots of IAM reports, CIS Benchmark config hardening, EC2, ECR, EKS, Spot termination, S3 access logging, KMS key rotation info, SSM, CloudTrail, CloudWatch billing alarm with SNS notification topic and subscription for email alerts
• GCP - Google Cloud Platform - massive GCP auto-inventory, scripts for GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform service account creation
• Kubernetes - massive Kubernetes auto-inventory, cluster management scripts & tricks
• Docker - Docker API, Dockerhub API, Quay.io API scripts
• Databases - fast CLI wrappers, instant Docker sandboxes (PostgreSQL, MySQL, MariaDB, SQLite), SQL scripts, SQL script testers against all versions of a DB, advanced .psqlrc
• Data - data tools, converters and format validators for Avro, Parquet, CSV, JSON, INI / Properties files (Java), LDAP LDIF, XML, YAML
• Big Data & NoSQL - Kafka, Hadoop, HDFS, Hive, Impala, ZooKeeper, Cloudera Manager API & Cloudera Navigator API scripts
• Git - GitHub, GitLab, Bitbucket, Azure DevOps - scripts for Git local & mirror management, GitHub, GitLab & BitBucket APIs
• Markdown - generate Markdown indexes and debug mdl issues like MD005 inconsistent list indentation in large README.md files
• CI/CD - Continuous Integration / Continuous Delivery - API scripts & build pipeline configs for most major CI systems:
  • Jenkins, Concourse, GoCD, TeamCity - one-touch boot & build
  • Azure DevOps Pipelines, GitHub Actions Workflows, GitLab CI, BitBucket Pipelines, AppVeyor, BuildKite, Travis CI, Circle CI, Codefresh, CodeShip, Drone.io, Semaphore CI, Shippable ...
  • Terraform Cloud, Octopus Deploy
  • Checkov / Bridgecrew Cloud
• AI & IPaaS - OpenAI (ChatGPT), Make.com
• Internet Services - Google Maps, Cloudflare, DataDog, Digital Ocean, Kong API Gateway, GitGuardian, Jira, NGrok, Traefik, Pingdom, Wordpress and various pastebins and file upload sites
• Java - Java utilies to debug running Java programs or decompile Java JAR code for deeper debugging
• Python - Python utilities & library management
• Perl - Perl utilities & library management
• Golang - Golang utilities
• Diagrams - scripts to generate diagrams from D2lang, MermaidJS and Python Mingrammer source code used in my HariSekhon/Diagrams-as-Code repo
• Media - video downloaders & converts, MP3 metadata editing, grouping and ordering of albums and audiobooks, mkv/avi to mp4 converters, 720p video downscaler for posting to social media, download YouTube videos or even entire channels and videos from other social media sites like Twitter / X or Facebook, terminal gif capture
• Spotify - 40+ Spotify API scripts for backups, managing playlists, track deduplication, URI conversion, search, add/delete, liked tracks, followed artists, top artists, top tracks etc.
• More Linux & Mac - more systems administration scripts, package installation automation
• Builds, Languages & Linting - programming language, build system & CI linting
• Templates - Templates for AWS, GCP, Terraform, Docker, Jenkins, Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml etc.
• Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

Dot Configs

Top-level dotfiles and configs/ directory:

• .* - dot conf files for lots of common software eg. advanced .vimrc, .gitconfig, massive .gitignore, .editorconfig, .screenrc, .tmux.conf etc.
  • .vimrc - contains many awesome vim tweaks, plus hotkeys for linting lots of different file types in place, including Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties files, LDAP LDIF etc without leaving the editor!
  • .screenrc - fancy screen configuration including advanced colour bar, large history, hotkey reloading, auto-blanking etc.
  • .tmux.conf - fancy tmux configuration include advanced colour bar and plugins, settings, hotkey reloading etc.
  • Git:
    • .gitconfig - advanced Git configuration
    • .gitignore - extensive Git ignore of trivial files you shouldn't commit
    • enhanced Git diffs
    • protections against committing AWS secret keys or merge conflict unresolved files

Bash Environment & Libraries

Top-level .bashrc and .bash.d/ directory:

• .bashrc - shell tuning and sourcing of .bash.d/*.sh
• .bash.d/*.sh - thousands of lines of advanced bashrc code, aliases, functions and environment variables for:
  • Linux & Mac
  • SCM - Git, Mercurial, Svn
  • AWS
  • GCP
  • Docker
  • Kubernetes
  • Kafka
  • Vagrant
  • automatic GPG and SSH agent handling for handling encrypted private keys without re-entering passwords, and lazy evaluation to only prompt key load the first time SSH is called
  • and lots more - see .bash.d/README for a more detailed list
  • run make bash to link .bashrc/.bash_profile and the .* dot config files to your $HOME directory to auto-inherit everything
• lib/*.sh - Bash utility libraries full of functions for Docker, environment, CI detection (Travis CI, Jenkins etc), port and HTTP url availability content checks etc. Sourced from all my other GitHub repos to make setting up Dockerized tests easier.

Installation Scripts

• install/install_*.sh - various simple to use installation scripts for common technologies like:
  • AWS CLI
  • Azure CLI
  • GCloud SDK
  • GitHub CLI
  • Terraform
  • Terragrunt
  • Direnv
  • Ansible
  • K3s
  • MiniKube (Kubernetes)
  • MiniShift (Redhat OpenShift / OKD dev VMs)
  • Maven
  • Gradle
  • SBT
  • EPEL
  • RPMforge
  • Homebrew
  • Travis CI
  • Circle CI
  • AppVeyor
  • BuildKite
  • Avro Tools
  • Parquet Tools
  • Prometheus
  • various JDKs and RDBMS JDBC connector jars
  • and many more...

Linux & Mac

bin/ directory:

• login.sh - logs to major Cloud platforms if their credentials are found in the environment, CLIs such as AWS, GCP, Azure, GitHub... Docker registries: DockerHub, GHCR, ECR, GCR, GAR, ACR, Gitlab, Quay...
• clean_caches.sh - cleans out OS package and programming language caches - useful to save space or reduce Docker image size
• command_return_to_current_window.sh - runs a Mac or Linux command that opens a window and then switches back to the original foreground window
• crypto_dice_rolls.sh - generates 100 random dice rolls to test a new crypto hardware wallet's fidelity (do not use this for your real crypto seed as your machine could be infected with malware which steals your seed phrase)
• delete_duplicate_files.sh - deletes duplicate files with (N) suffixes, commonly caused by web browser downloads, in the given or current directory. Checks they're exact duplicates of a matching basename file without the (N) suffix with the exact same checksum for safety. Prompts to delete per file. To auto-accept deletions, do yes | delete_duplicate_files.sh. This is a fast way of cleaning up your ~/Downloads directory and can be put your user crontab
• diff_line_threshold.sh - compares two files vs a line count diff threshold to determine if they are radically different. Used to avoid overwriting files which are not mere updates but completely different files
• disk_speed_read_sequential_dd.sh - runs a sequential read speed test from the given file using dd and bypassing filesystem cache for a more accurate test
• disk_speed_read_random_dd.sh - runs a random I/O read speed test from the given file using dd and bypassing filesystem cache for a more accurate test
• disk_speed_write_sequential_dd.sh - runs a sequential write speed test to a file in the given or current directory using dd and bypassing filesystem cache for a more accurate test
• disk_speed_read_sequential_fio.sh - runs a sequential read speed test in the current or given directory using fio
• disk_speed_read_random_fio.sh - runs a random I/O read test in the current or given directory using fio
• disk_speed_write_sequential_fio.sh - runs a sequential write speed test to the current or given directory using fio
• disk_speed_write_random_fio.sh - runs a sequential write speed test to the current or given directory using fio
• download_url_file.sh - downloads a file from a URL using wget with no clobber and continue support, or curl with atomic replacement to avoid race conditions. Used by github/github_download_release_file.sh, github_download_release_jar.sh, and install/download_*_jar.sh
• curl_auth.sh - shortens curl command by auto-loading your OAuth2 / JWT API token or username & password from environment variables or interactive starred password prompt through a ram file descriptor to avoid placing them on the command line (which would expose your credentials in the process list or OS audit log files). Used by many other adjacent API querying scripts
• curl_with_cookies.sh - extracts cookies for a given URL from your \$BROWSER's cookie jar and passes them to the curl command along with the rest of the args (workaround for older curl builds and Homebrew builds that don't have the newer `--cookies-from-browser functionality)
• emptytrash.sh - empties the Trash portably on Linux and Mac. On Mac, combine with a post-run of mac_delete_local_snapshots.sh to ensure space is reclaimed
• find_duplicate_files_by_*.sh / - finds duplicate files by size and/or checksum in given directory trees. Checksums are only done on files that already have matching byte counts for efficiency
• find_broken_links.sh - find broken links with delays to avoid tripping defenses
• find_broken_symlinks.sh - find broken symlinks pointing to non-existent files/directories
• find_lock.sh - tries to find if a lockfile is used in the given or current working directory by taking snapshots of the file list before and after a prompt in which you should open/close an application
• foreach_path_bin.sh - runs each binary of the given name found in $PATH with the args given. Useful to find all the installed versions of a program in different paths eg. ~/bin/ vs /usr/local/bin/ eg. foreach_path_bin.sh terraform --version
• http_duplicate_urls.sh - find duplicate URLs in a given web page
• htmldecode.sh - decodes HTML encoding. Detects available tools such as Perl, Python or xmlstarlet and uses whatever is available
• html_hrefs.sh - parses href links out of an HTML page given as a file or read from stdin. See also urlextract.sh for a simpler pure bash HTTP(s) URL onlyonly string extractor
• ldapsearch.sh - shortens ldapsearch command by inferring switches from environment variables
• ldap_user_recurse.sh / ldap_group_recurse.sh - recurse Active Directory LDAP users upwards to find all parent groups, or groups downwards to find all nested users (useful for debugging LDAP integration and group-based permissions)
• linux_distro_versions.sh - quickly returns the list of major versions for a given Linux distro
• linux_command_return_to_current_window.sh - runs a Linux command that opens a window and then switches back to the original foreground window
• lockdir.sh - creates a portable atomic lock using a given directory. Useful for a script to prevent two instances from running at the same time to avoid either data corruption or API rate limits. If you're on Linux you should use flock if available for its enhanced kernel integration
• mac_command_return_to_current_window.sh - runs a Mac command that opens a window and then switches back to the original foreground window
• mv.sh - moves directory trees resumably and removes the source files as they're copied over. Useful to migrate data from one disk to another, optionally with checksums. Uses rsync and shows the overall % of files transferred and the MB/s data transfer rate
• network_gateway.sh - get the network gateway IP address on Linux or Mac
• network_randomize_mac_address.sh - assigns a new random mac address to your network interface
• open.sh - opens given arg, file or URL using whatever default system opener is available for Linux or Mac
• organize_downloads.sh - moves files of well-known extensions in the $HOME/Downloads directory older than 1 week to capitalized subdirectories of their type to keep the $HOME/Downloads/ directory tidy
• copy_to_clipboard.sh - copies stdin or string arg to system clipboard on Linux or Mac
• paste_from_clipboard.sh - pastes from system clipboard to stdout on Linux or Mac
• paste_from_clipboard_upon_changes.sh - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes
• paste_diff_settings.sh - takes snapshots of before and after clipboard changes and diffs them to show config changes. Useful to figure out how to backport UI generated changes that are exportable to a global Config-as-Code such as in Jenkins JCasC
• processes_ram_sum.sh - sums the RAM usage of all processes matching a given regex in GB to one decimal place
• pldd.sh - parses /proc on Linux to show the runtime .so loaded dynamic shared libraries a program pid is using. Runtime equivalent of the classic static ldd command and because the system pldd command often fails to attach to a process
• random_select.sh - selects one of given args at random. Useful for sampling, running randomized subsets of large test suites etc.
• random_number.sh - prints a random integer between two integer arguments (inclusive)
• random_string.sh - prints a random alphanumeric string of a given length
• screen_buffer_to_stdout.sh - dumps the GNU Screen copy buffer contents to stdout
• screen_buffer_to_clipboard.sh - dumps the GNU Screen copy buffercontents to a temp file and copies to clipboard for sharing & debugging purposes
• screen_terminal_to_stdout.sh - dumps the GNU Screen terminal contents to stdout
• screen_terminal_to_clipboard.sh - dumps the GNU Screen terminal contents to a temp file and copies to clipboard for sharing & debugging purposes
• shields_embed_logo.sh - base64 encodes a given icon file or url and prints the logo=... url parameter you need to add the shields.io badge url
• shorten_clipboard_text.sh - shortens the text in clipboard to make social media comments shorter. Replaces and with & and crushes out multiple blank lines. I used this for LinkedIn comments due to the short 1250 character limit
• shorten_text_selection.sh - shortens the selected text in the prior window. Faster way to use the above without having to copy and paste clipboard manually
• shred_file.sh - overwrites a file 7 times to DoD standards before deleting it to prevent recovery of sensitive information
• shred_free_space.sh - overwrites free space to prevent recovery of sensitive information for files that have already been deleted
• sleepuntil.sh - sleep until the given HH:MM clock time. Useful for delaying the run of a foreground terminal program to inherit your terminal's environment variables (which may contain API keys) or interactively manage a process in your terminal, which cannot be done with the at command. I use this for my HariSekhon/Spotify-Playlists backup each night so it waits in terminal for me to arrive home and runs it interactively to allow human gate safety approvals before Git committing any song removals from any playlist
• split.sh - split large files into N parts (defaults to the number of your CPU cores) to parallelize operations on them
• ssl_get_cert.sh - gets a remote host:port server's SSL cert in a format you can pipe, save and use locally, for example in Java truststores
• ssl_verify_cert.sh - verifies a remote SSL certificate (battle tested more feature-rich version check_ssl_cert.pl exists in the Advanced Nagios Plugins repo)
• ssl_verify_cert_by_ip.sh - verifies SSL certificates on specific IP addresses, useful to test SSL source addresses for CDNs, such as Cloudflare Proxied sources before enabling SSL Full-Strict Mode for end-to-end, or Kubernetes ingresses (see also curl_k8s_ingress.sh)
• text_filter_ending_substrings.sh - for a given patterns file of substring endings, print all lines that match in the following files. Uses awk to safely handle all characters as literals, unlike grep, while also maintaining end anchoring which you cannot do using grep -F. Optimized awk code uses a bucketing hash for performance to not attempt matching lines which are shorter than patterns, reducing the number of match attempts
• tmux_vertical.sh - launches tmux with N-way vertical shell split or commands given as args in equally balanced vertical panes. Fast way to launch a bunch of shell or commands in an easily reviewable side-by-side way
• tmux_horizontal.sh - same as above but split horizontally
• tmux_square.sh - same as above but with 4 panes in a square tiled view
• urlencode.sh / urldecode.sh - URL encode/decode quickly on the command line, in pipes etc.
• urlextract.sh - extracts the URLs from a given string arg, file or standard input
• url_extract_redirects.sh - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the redirected urls instead to stdout
• url_replace_redirects.sh - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the entire contents to stdout with the urls replaced by the redirected urls
• urlopen.sh - opens the URL given as an arg, or first URL found from stdin or a given file. Uses the system's default browser
• vagrant_hosts.sh - generate /etc/hosts output from a Vagrantfile
• vagrant_total_mb.sh - calculate the RAM committed to VMs in a Vagrantfile

See also Knowledge Base notes for Linux and Mac.

Mac & AppleScript

Mac automation scripts to automate the Mac UI and settings

bin/ directory:

• emptytrash.sh - empties the Trash portably on Linux and Mac. On Mac, combine with a post-run of mac_delete_local_snapshots.sh to ensure space is reclaimed
• mac_diff_settings.sh - takes before and after snapshots of UI setting changes and diffs them to make it easy to find defaults keys to add to setup/mac_settings.sh to save settings
• mac_restore_file.sh - checks all the backup mount points for the latest backup that has a given file and then restores it
• mac_backup_du_in_progress.sh - find large files in the currently in-progress Time Machine backup to find out what is taking so long and racking up so many more GB of changes than you expect. This helps discover large but unnecessary files that you might want to exclude using the adjacent script mac_backup_exclude_paths.sh
• mac_backup_exclude_paths.sh - excludes many common large caches, docker and VM paths from macOS Time Machine backups
• mac_backup_find_excluded_paths.sh - does a deep search for macOS Time Machine excluded backup paths on file/folder attributes. See HariSekhon/Knowledge-Base Mac page for why
• mac_command_return_to_current_window.sh - runs a Mac command that opens a window and then switches back to the original foreground window
• mac_rmdir.sh - safely delete a directory on Mac only if it is empty of actual data, by first removing macOS hidden metadata files and dirs such as .fseventsd/, .Spotlight-V100/ and .DS_Store - straight rmdir fails otherwise
• mac_iso_to_usb.sh - converts a given ISO file to a USB bootable image and burns it onto a given or detected inserted USB drive
• mac_ramdisk.sh - creates a mac ramdisk of given MB size. Useful for performance, or even testing disk write scripts such as disk_speed_write_*.sh without wearing out your SSD
• mac_delete_local_snapshots.sh - deletes local macOS snapshots to free up disk space. When there is a substantial discrepancy between what the df -h command and the Finder UI shows, this is often the cause. Recommended to first run emptytrash.sh
• copy_to_clipboard.sh - copies stdin or string arg to system clipboard on Linux or Mac
• paste_from_clipboard.sh - pastes from system clipboard to stdout on Linux or Mac
• paste_from_clipboard_upon_changes.sh - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes
• paste_diff_settings.sh - takes snapshots of before and after clipboard changes and diffs them to show config changes. Useful to figure out how to backport UI generated changes that are exportable to a global Config-as-Code such as in Jenkins JCasC

applescript/ directory:

• keystrokes.sh - send N keystroke combinations
• mouse_clicks.sh - send N mouse click combinations to sequence of screen coordinates
  • get_mouse_coordinates.sh - print the current mouse coordinates - to know what to pass to above script
• mouse_clicks_remote_desktop.sh - switches to Microsoft Remote Desktop, waits 10 seconds and then clicks the mouse once a minute to prevent the screensaver from coming on. Workaround to Active Directory Group Policies that don't let you disable the screensaver. Point your mouse to an area that will have no mouse click effect, the Cmd-Tab to Terminal and run this
• get_frontmost_process_title.applescript - detect the frontmost window
  • to detect if you should send keystrokes / mouse clicks)
• set_frontmost_process.applescript - switch to bring the given app to the foreground to send keystrokes / mouse clicks to it
  • browser_get_default.applescript - get the default configured browser in format passable to Applescript (for above script)
• is_screen_locked.py - detect if the screen is locked to stop sending keystrokes or mouse clicks
• is_screensaver_running.applescript - detect if the screensaver is running to stop sending keystrokes or mouse clicks
• reopen_app.sh - relaunch a given app (used to reload Shazam to detect DB changes after removing tracks programmatically from its DB)
• spotify_app_search.sh - runs a search in the Spotify App on Mac using Applescript
• spotify_app_search_filelist.sh - searches the Spotify App for each track in a file, prompting to continue to the next track
• shazam_app_dump_tracks.sh - dumps artist - track one per line from the Shazam local sqlite DB
• shazam_app_delete_track.sh - deletes a given "artist" "track" from the Shazam local sqlite DB
• shazam_search_spotify_then_delete_track.sh - searches for each Shazam'd track in the local Spotify desktop app, then prompts to delete each track from the local Shazam DB once you've saved it in Spotify. Useful to migrate Shazam'd tracks to Spotify after Apple removed the integration
• screensaver_activate.applescript - activate screensaver
• shorten_text_selection.applescript - shortens the selected text in the prior window. Replaces and with & and crushes out multiple blank lines. I use this for LinkedIn comments due to the short 1250 character limit
• start_app_at_login.sh - adds an app to the Login items to auto-start

Hammerspoon code has been moved to its own repo:

See also Mac page in HariSekhon/Knowledge-Base.

Monitoring

monitoring/ directory:

• dump_stats.sh - dumps common command outputs to text files in a local tarball. Useful to collect support information for vendor support cases
• grafana_api.sh - queries the Grafana API with authentication
• log_timestamp_large_intervals.sh - finds log lines whose timestamp intervals exceed the given number of seconds and outputs those log lines with the difference between the last and current timestamps. Useful to find actions that are taking a long time from log files such as CI/CD logs
• prometheus.sh - starts Prometheus locally, downloading it if not found in $PATH
• prometheus_docker.sh - starts Prometheus in Docker using docker-compose
• prometheus_node_exporter.sh - starts Prometheus node_exporter locally, downloading it if not found in $PATH
• ssh_dump_stats.sh - uses SSH and dump_stats.sh to dump common command outputs from remote servers to a local tarball. Useful for vendor support cases
• ssh_dump_logs.sh - Uses SSH to dump logs from server to local text files for uploading to vendor support cases

See doc pages in HariSekhon/Knowledge-Base on Grafana, Prometheus, OpenTSDB, InfluxDB etc.

Databases

mysql/, postgres/, sql/ and bin/ directories:

• sql/ - 100+ SQL scripts for PostgreSQL, MySQL, Google BigQuery and AWS Athena CloudTrail logs integration
• sqlite.sh - one-touch SQLite, starts sqlite3 shell with sample 'chinook' database loaded
• mysql*.sh - MySQL scripts:
  • mysql.sh - shortens mysql command to connect to MySQL by auto-populating switches from both standard environment variables like $MYSQL_TCP_PORT, $DBI_USER, $MYSQL_PWD (see doc) and other common environment variables like $MYSQL_HOST / $HOST, $MYSQL_USER / $USER, $MYSQL_PASSWORD / $PASSWORD, $MYSQL_DATABASE / $DATABASE
  • mysql_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
  • mysql_*.sh - various scripts using mysql.sh for row counts, iterating each table, or outputting clean lists of databases and tables for quick scripting
  • mysqld.sh - one-touch MySQL, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
  • see also the SQL Scripts repo for many more straight MySQL SQL scripts
• mariadb.sh - one-touch MariaDB, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
• postgres*.sh / psql.sh - PostgreSQL scripts:
  • postgres.sh - one-touch PostgreSQL, boots docker container + drops in to psql shell, with /sql scripts mounted in container for easy sourcing eg. \i /sql/<name>.sql. Optionally loads sample 'chinook' database
  • psql.sh - shortens psql command to connect to PostreSQL by auto-populating switches from environment variables, using both standard postgres supported environment variables like $PG* (see doc) as well as other common environment variables like $POSTGRESQL_HOST / $POSTGRES_HOST / $HOST, $POSTGRESQL_USER / $POSTGRES_USER / $USER, $POSTGRESQL_PASSWORD / $POSTGRES_PASSWORD / $PASSWORD, $POSTGRESQL_DATABASE / $POSTGRES_DATABASE / $DATABASE
  • postgres_foreach_table.sh - executes a SQL query against every table, replacing {db}, {schema} and {table} in each iteration eg. select count(*) from {table}
  • postgres_*.sh - various scripts using psql.sh for row counts, iterating each table, or outputting clean lists of databases, schemas and tables for quick scripting
  • checks/check_sqlfluff.sh - recursively iterates all SQL code files found in the given or current directory and runs SQLFluff linter against them, inferring the different SQL dialects from each path/filename/extension

AWS - Amazon Web Services

aws/ directory:

• AWS scripts - aws_*.sh:
  • aws_profile.sh - switches to an AWS Profile selected from a convenient interactive menu list of AWS profiles from $AWS_CONFIG_FILE - useful when you have lots of AWS work profiles
    • see also HariSekhon/Environments for automated switching using direnv when cding into relevant directories
  • aws_cli_create_credential.sh - creates an AWS service account user for CI/CD or CLI with Admin permissions (or other group or policy), creates an AWS Access Key, saves a credentials CSV and even prints the shell export commands and aws credentials file config to configure your environment to start using it. Useful trick to avoid CLI reauth to aws sso login every day.
  • aws_terraform_create_credential.sh - creates a AWS terraform service account with Administrator permissions for Terraform Cloud or other CI/CD systems to run Terraform plan and apply, since no CI/CD systems can work with AWS SSO workflows. Stores the access key as both CSV and prints shell export commands and credentials file config as above
  • .envrc-aws - copy to .envrc for direnv to auto-load AWS configuration settings such as AWS Profile, Compute Region, EKS cluster kubectl context etc.
    • calls .envrc-kubernetes to set the kubectl context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global ~/.kube/config context
  • aws_sso_ssh.sh - launches local AWS SSO authentication pop-up (if not already authenticated), then scp's the latest resultant ~/.aws/sso/cache/ file to the remote server and SSH's there so that you can use AWS CLI or kubectl to EKS remotely on that server easily, without having to copy and paste the token from remote aws sso login to your local web browser
  • aws_terraform_create_s3_bucket.sh - creates a Terraform S3 bucket for storing the backend state, locks out public access, enables versioning, encryption, and locks out Power Users role and optionally any given user/group/role ARNs via a bucket policy for safety
  • aws_terraform_create_dynamodb_table.sh - creates a Terraform locking table in DynamoDB for use with the S3 backend, plus custom IAM policy which can be applied to less privileged accounts
  • aws_terraform_create_all.sh - runs all of the above, plus also applies the custom DynamoDB IAM policy to the user to ensure if the account is less privileged it can still get the Terraform lock (useful for GitHub Actions environment secret for a read only user to generate Terraform Plans in Pull Request without needing approval)
  • aws_terraform_iam_grant_s3_dynamodb.sh - creates IAM policies to access any S3 buckets and DynamoDB tables with terraform-state or tf-state in their names, and attaches them to the given user. Useful for limited permissions CI/CD accounts that run Terraform Plan eg. in GitHub Actions pull requests
  • aws_account_summary.sh - prints AWS account summary in key = value pairs for easy viewing / grepping of things like AccountMFAEnabled, AccountAccessKeysPresent, useful for checking whether the root account has MFA enabled and no access keys, comparing number of users vs number of MFA devices etc. (see also check_aws_root_account.py in Advanced Nagios Plugins)
  • aws_billing_alarm.sh - creates a CloudWatch billing alarm and SNS topic with subscription to email you when you incur charges above a given threshold. This is often the first thing you want to do on an account
  • aws_budget_alarm.sh - creates an AWS Budgets billing alarm and SNS topic with subscription to email you when both when you start incurring forecasted charges of over 80% of your budget, and 90% actual usage. This is often the first thing you want to do on an account
  • aws_batch_stale_jobs.sh - lists AWS Batch jobs that are older than N hours in a given queue
  • aws_batch_kill_stale_jobs.sh - finds and kills AWS Batch jobs that are older than N hours in a given queue
  • aws_cloudfront_distribution_for_origin.sh - returns the AWS CloudFront ARN of the distribution which serves origins containing a given substring. Useful for quickly finding the CloudFront ARN needed to give permissions to a private S3 bucket exposed via CloudFront
  • aws_cloudtrails_cloudwatch.sh - lists Cloud Trails and their last delivery to CloudWatch Logs (should be recent)
  • aws_cloudtrails_event_selectors.sh - lists Cloud Trails and their event selectors to check each one has at least one event sele