Table of Contents

Open all
Close all
Preface
23
Bash, PowerShell, or Python?
23
About This Book
24
Scripting and AI Tools
24
Scripting as a Core Competence for Efficient IT Work
25
PART I Scripting Languages
27
1 Scripting: Doing One Thing
29
1.1 What Does Scripting Mean?
29
1.1.1 Scripting versus Programming
30
1.1.2 Glue Languages and Glue Code
30
1.1.3 Do One Thing and Do It Well
31
1.2 Scripting Languages
31
1.2.1 Bash and Zsh
32
1.2.2 PowerShell
33
1.2.3 Python
34
1.2.4 Many Similarities, Even More Differences
35
1.3 The Agony of Choice
35
2 Ten Times Ten Lines
39
2.1 Markdown Spell Checker (Bash)
39
2.2 Sorting Images by Date (PowerShell)
40
2.3 Converting a JSON File to XML Format (Python)
41
2.4 Daily Server Backups (Bash)
42
2.5 Web Scraping (Python)
43
2.6 Logging Weather Data (Python)
44
2.7 Microsoft Hyper-V Cleanup (PowerShell)
45
2.8 Statistical Analysis of a Logging File (Bash)
46
2.9 Uploading Files to the Cloud (PowerShell)
46
2.10 Cloning Virtual Machines (Bash)
47
3 Bash and Zsh
49
3.1 Terminal, Shell, and Bash
49
3.1.1 Determining the Current Shell
50
3.2 Installation
50
3.2.1 Using Bash in a Virtual Machine
51
3.2.2 Windows Subsystem for Linux
51
3.2.3 Git Bash
53
3.2.4 Docker
53
3.2.5 Bash Configuration (/etc/profile and .bashrc)
53
3.3 Running Commands Interactively
53
3.3.1 Operating the Terminal
55
3.3.2 Online Help
56
3.3.3 Command Reference
56
3.4 Zsh as an Alternative to Bash
56
3.4.1 Installation
57
3.4.2 Zsh Configuration (/etc/zshrc and .zshrc)
57
3.4.3 Oh My Zsh
58
3.5 The First Bash Script
58
3.5.1 Hash Bang (Shebang)
58
3.5.2 Shebang Variants
59
3.5.3 Making the Script Executable (chmod +x)
60
3.5.4 Hello, World!
60
3.5.5 Elementary Syntax Rules
61
3.5.6 A “Real” Example: Backup Script
61
3.5.7 Running Scripts without ./
63
3.6 Running Commands
63
3.6.1 Serial Execution
64
3.6.2 Conditional Command Execution
64
3.6.3 Running Commands in the Background
65
3.6.4 Running Commands in a Subshell
65
3.7 Standard Input and Standard Output
66
3.7.1 Redirecting Input and Output
66
3.7.2 The Pipe Operator
68
3.7.3 Redirecting and Displaying Outputs at the Same Time
68
3.8 Globbing, Brace Extension, and Handling File and Directory Names
68
3.8.1 Recursive Globbing
70
3.8.2 Access to Important Directories
70
3.8.3 Brace Extension
70
3.8.4 Filenames and Directory Names with Spaces
71
3.9 Variables
72
3.9.1 Initializing and Deleting Variables
72
3.9.2 Declaring Variables
73
3.9.3 Performing Calculations in Bash
73
3.9.4 Arrays
74
3.9.5 Predefined Variables
76
3.9.6 Environment Variables
76
3.10 Strings
77
3.10.1 Single versus Double Quotation Marks
77
3.10.2 Outputting Strings (echo)
78
3.10.3 Colors
79
3.10.4 Entering or Reading Character Strings (read)
80
3.10.5 Substitution and Expansion Mechanisms
81
3.10.6 Parameter Substitution
81
3.10.7 Heredocs and Herestrings
84
3.10.8 The Backslash
85
3.11 Branches
85
3.11.1 if Short Notation with && or ||
86
3.11.2 Conditions
87
3.11.3 Branches with case
89
3.11.4 Parameter Analysis via case
90
3.12 Loops
91
3.12.1 Processing Filenames with Spaces
93
3.12.2 while and until
93
3.12.3 Loops over Text Files
94
3.13 Functions
95
3.13.1 Local Variables
96
3.14 Error Protection
96
3.14.1 Detecting Errors
96
3.14.2 Canceling in Case of Errors
97
3.14.3 exit
97
3.14.4 Responding to Signals (trap)
98
3.14.5 Timeout
98
4 PowerShell
99
4.1 Installation
99
4.1.1 Installation on Linux
101
4.1.2 Installation on macOS
102
4.1.3 Limitations of PowerShell on Windows versus Linux/macOS
102
4.1.4 Configuration
103
4.2 Windows Terminal
103
4.2.1 Configuration
104
4.2.2 Operation
105
4.3 Calling cmdlets and Functions
106
4.3.1 cmdlets
108
4.3.2 The Verb-Noun Terminology
108
4.3.3 Aliases
108
4.3.4 Parameters and Options
109
4.3.5 Splatting
109
4.3.6 Functions
110
4.3.7 Running Conventional Commands
110
4.3.8 Online Help
111
4.4 Combining Commands
112
4.4.1 Chains of Commands
113
4.4.2 Multiline Statements
114
4.4.3 Dealing with Special Characters
114
4.5 The First Script
114
4.5.1 Hello, World!
115
4.5.2 Trouble with the Execution Policy
116
4.5.3 Setting Up a Custom Script Directory
117
4.5.4 Running Scripts on Linux and macOS
118
4.5.5 Example: Cleaning Up the Downloads Directory
118
4.5.6 Differentiating Between PowerShell Versions
120
4.6 Variables, Strings, and Objects
120
4.6.1 Data Types
121
4.6.2 Calculating and Comparing
123
4.6.3 Strings
124
4.6.4 Command Substitution
126
4.6.5 Objects
127
4.6.6 Assembling Custom Objects
128
4.6.7 Predefined Variables
129
4.6.8 Environment Variables
130
4.7 Arrays and Hash Tables
131
4.7.1 Creating and Processing Custom Arrays
131
4.7.2 Hash Tables
133
4.8 Output Redirection
133
4.8.1 Streams
133
4.8.2 Output Redirection
134
4.8.3 Duplicating Output (Tee-Object)
136
4.8.4 (Not) Formatting the Output
136
4.8.5 No Input Redirection
136
4.9 Loops
137
4.9.1 ForEach-Object
138
4.9.2 for, while, do-while, and do-until
138
4.10 Branches
139
4.10.1 Case Distinctions Using switch
141
4.11 Functions and Parameters
141
4.11.1 Function Results
142
4.11.2 return
143
4.11.3 Parameters
143
4.11.4 Calling Functions
144
4.11.5 Processing Standard Input
145
4.11.6 Syntax Variants
145
4.12 Modules
147
4.12.1 Module Directories
147
4.12.2 Modules for Experts
148
4.13 Error Protection
149
4.13.1 Set-StrictMode
150
4.13.2 Aborting Scripts on Errors
150
4.13.3 Separating Error Messages from Other Output
150
4.13.4 Error Protection Using try/catch
151
4.13.5 Terminating the Script Execution Prematurely (Exit)
152
5 Python
153
5.1 Installing Python
153
5.1.1 Windows
154
5.1.2 macOS
155
5.2 Getting to Know Python in a Terminal Window
156
5.3 Programming Custom Scripts
157
5.3.1 Shebang
157
5.4 Elementary Syntax Rules
158
5.4.1 Block Elements
159
5.4.2 Comments
160
5.5 Numbers
160
5.5.1 Floats
161
5.5.2 Random Numbers
161
5.6 Strings
162
5.6.1 Raw Strings
163
5.6.2 Processing Strings
163
5.6.3 Slicing
165
5.6.4 print and input
166
5.6.5 Formatting Outputs
167
5.7 Lists
169
5.7.1 List Comprehension
170
5.7.2 map and filter
170
5.8 Tuples, Sets, and Dictionaries
171
5.8.1 Sets
172
5.8.2 Dictionaries
173
5.9 Variables
174
5.9.1 Assignments
175
5.9.2 Data Types
176
5.9.3 Mutable or Immutable?
177
5.10 Operators
178
5.10.1 Combining Assignment and Evaluation
180
5.11 Branches (if)
180
5.11.1 Conditions
181
5.11.2 Short Notation with if
181
5.12 Loops (for and while)
181
5.12.1 break, continue, and else
182
5.12.2 Loops over Number Ranges (range)
183
5.12.3 Loops over the Characters of a String
183
5.12.4 Loops over Lists, Tuples, and Sets
184
5.12.5 Loops over Dictionaries
185
5.12.6 Loops over the Parameters of a Script
185
5.12.7 Globbing on Windows
186
5.12.8 Loops over the Lines of a Text File
186
5.12.9 Loops over All Files in a Directory
187
5.13 Functions
187
5.13.1 Global and Local Variables
188
5.13.2 Parameters
189
5.13.3 Optional Parameters
190
5.13.4 Variable Number of Parameters
190
5.13.5 Lambda Functions
191
5.14 Processing Text Files
192
5.14.1 Example: Analyzing a CSV File
193
5.14.2 Example: Creating SQL Commands
194
5.15 Error Protection
195
5.15.1 try/except
195
5.16 System Functions
195
5.16.1 Access to the Program Parameters
196
5.16.2 Access to Standard Input and Standard Output
196
5.16.3 Exiting a Program
196
5.16.4 Calling Linux Commands
196
5.16.5 Processing Results
197
5.16.6 Running Commands through the Shell
197
5.16.7 Error during a Command Call
197
5.16.8 Waiting (sleep)
198
5.17 Modules
198
5.17.1 import
198
5.17.2 Distributing Custom Scripts across Multiple Files
199
5.18 Installing Additional Modules Using pip
199
5.18.1 Installing pip
200
5.18.2 Applying pip
200
5.18.3 pip Problems with Windows
200
5.18.4 pip Problems with Linux
201
5.18.5 requirements.txt
203
5.18.6 pipenv
203
PART II Work Techniques and Tools
205
6 Linux Toolbox
207
6.1 Directories and Files
207
6.1.1 Listing, Copying, Moving, and Deleting Files
208
6.2 Finding Files
209
6.2.1 Text Search Using grep
210
6.3 Compressing and Archiving Files
210
6.3.1 Archiving Files Using tar
211
6.3.2 ZIP Files
212
6.4 Using Root Privileges
212
6.4.1 sudo Privileges for Selected Accounts Only
213
6.4.2 Issues with Output Redirection
213
6.4.3 Access Rights
214
6.4.4 Managing Processes
215
6.4.5 Terminating Processes
215
6.4.6 Background Processes and System Services
216
6.4.7 Logging Files
216
6.4.8 Determining Free Memory
217
6.4.9 Determining Other System Information
218
6.5 Software Installation
218
6.5.1 Updating Software
219
6.5.2 Installing Additional Packages
219
6.6 Other Commands
220
7 cmdlets for PowerShell
223
7.1 Directories and Files
223
7.1.1 Listing, Copying, Moving, and Deleting Files
224
7.1.2 Reading and Writing Text
225
7.1.3 Inputting and Outputting Text
226
7.2 Finding Files
227
7.2.1 Searching Text Files
228
7.2.2 Text versus Objects
229
7.3 Compressing and Archiving Files
230
7.3.1 Compressing Search Results
231
7.4 Process Management
232
7.4.1 Launching Other PowerShell Scripts
233
7.4.2 Running Commands in the Background
233
7.4.3 Managing Services
233
7.4.4 Working on Other Computers
234
7.5 Registration Database and System Information
235
7.5.1 Determining System Information
237
7.5.2 Evaluating the Logging System
238
7.6 Processing cmdlet Results
238
7.6.1 Select-Object and Sort-Object
239
7.6.2 Where-Object
239
7.6.3 Group-Object
240
7.6.4 ForEach-Item
240
7.6.5 Measure-Object
240
7.6.6 Formatting and Exporting cmdlet Results
241
7.7 Other cmdlets
243
7.8 Installing Additional Modules
243
7.8.1 Listing Modules, Commands, and Package Sources
244
7.8.2 NuGet and winget
245
7.9 Standard Aliases
246
8 Analyzing Texts with Filters and Pipes
249
8.1 grep, sort, cut, and uniq
249
8.1.1 grep
250
8.1.2 wc
251
8.1.3 cut
252
8.1.4 sort
252
8.1.5 head and tail
253
8.1.6 uniq
254
8.1.7 tr
255
8.1.8 awk and sed
255
8.2 Example: Statistical Data Analysis
256
8.2.1 Data Extraction via Script
257
8.3 Example: ping Analysis
258
8.3.1 ping Call via Script
259
8.4 Example: Apache Log Analysis
260
8.4.1 Extracting IP Addresses
261
8.4.2 Identifying Popular Pages
262
8.5 CSV Files
263
8.5.1 Processing CSV Files Using Python
264
8.5.2 Processing CSV Files in PowerShell
265
9 Regular Expressions
267
9.1 Syntax Rules for Regular Expressions
268
9.1.1 Characters
268
9.2 Groups and Alternatives
270
9.2.1 Quantifiers
271
9.2.2 On Greed (Greedy versus Lazy)
272
9.2.3 Alpha and Omega
273
9.2.4 Exercise Pages and Cheat Sheets
274
9.2.5 Example: Recognizing a Date
275
9.2.6 Example: Recognizing IPv4 Addresses
275
9.2.7 Example: Recognizing Email Addresses
276
9.3 Regular Expressions in Bash (grep, sed)
276
9.3.1 Bash Comparison Operator for Regular Expressions
276
9.3.2 Filtering Text Using grep
277
9.3.3 The "sed" Stream Editor
278
9.3.4 Searching and Replacing Using sed
279
9.3.5 sed Example: Manipulating Paths of Image Files
279
9.4 Regular Expressions in PowerShell
280
9.4.1 The match and replace Operators
280
9.4.2 Select-String
281
9.4.3 Example: Input Validation
281
9.4.4 Example: List of Figures
282
9.5 Regular Expressions in Python (re Module)
283
9.5.1 Example: Verifying a MAC Address
284
9.5.2 Example: Anonymizing a Logging File
285
10 JSON, XML, and INI
287
10.1 JSON in PowerShell
287
10.1.1 Example: Saving Event Log Entries in JSON Format
288
10.1.2 Example: Analyzing Domain Queries
288
10.1.3 Example: Converting between CSV and JSON
289
10.2 JSON and Python
290
10.2.1 Example: Collecting Birthdays
291
10.2.2 Example: Determining Holidays
292
10.3 JSON in Bash
295
10.3.1 Viewing JSON Files Interactively Using fx
297
10.4 XML in PowerShell
298
10.4.1 XML Data Type
298
10.4.2 Select-Xml
299
10.4.3 ConvertTo-Xml, Export-Clixml, and Import-Clixml
301
10.5 XML and Python
302
10.5.1 Example: Creating a Dictionary for Country Codes
303
10.5.2 Example: Analyzing an RSS Feed
304
10.5.3 Example: Extracting MAC Addresses from Virtual Machine Files
305
10.6 XML in Bash
305
10.6.1 xmllint
306
10.6.2 XMLStarlet
307
10.7 INI Files
307
10.7.1 Python
308
10.7.2 PowerShell
308
10.7.3 Bash
308
11 Running Scripts Automatically
309
11.1 cron
309
11.1.1 /etc/crontab
310
11.1.2 Personal crontab Files
311
11.1.3 Hourly, Daily, Weekly, and Monthly Directories
312
11.1.4 Alternatives to cron
312
11.1.5 Starting Jobs Automatically on macOS
312
11.2 Example: Web Server Monitoring
313
11.2.1 Testing and Troubleshooting
314
11.2.2 Real Monitoring
314
11.3 Microsoft Windows Task Scheduler
315
11.3.1 Troubleshooting
317
11.3.2 Setting Up Tasks via cmdlets
318
11.4 Example: Saving Exchange Rates
319
11.5 Tracking File System Changes
320
11.5.1 inotify
321
11.5.2 Alternatives to inotify
322
12 SSH
323
12.1 Installing the SSH Client and Server
323
12.1.1 Linux
324
12.1.2 macOS
324
12.1.3 Windows
324
12.1.4 Using an Editor in an SSH Session
326
12.1.5 Securing an SSH Server
327
12.2 Working with SSH
327
12.2.1 Host Verification
328
12.2.2 Potential Issues and Their Cause
328
12.2.3 Running Linux and macOS Commands
329
12.2.4 Running Windows Commands
330
12.2.5 SSH Remoting in PowerShell
330
12.3 scp and rsync
332
12.3.1 Copy-Item with SSH Remoting
332
12.3.2 rsync
333
12.4 SSH Authentication with Keys
333
12.4.1 Generating a Key Pair
334
12.4.2 Storing the Public Component of the Key on the Server (macOS and Linux)
335
12.4.3 Storing the Public Component of the Key on the Server (Windows)
335
12.5 Example: Uploading Images to a Linux Web Server
336
12.5.1 Preparatory Measures
336
12.5.2 Bash Script
337
12.5.3 PowerShell Script
338
12.6 Example: Analyzing Virtual Machines
339
13 Visual Studio Code
341
13.1 Introduction
341
13.1.1 Comparing VS Code, VSCodium, and Visual Studio
342
13.1.2 Think in Terms of Directories, Not Files!
342
13.2 Language-Specific VS Code Extensions
343
13.2.1 PowerShell Extension
343
13.2.2 Python Extension
343
13.2.3 Bash/Shell Extensions
344
13.3 Remote – SSH Extension
345
13.3.1 Applying the Remote – SSH Extension
345
13.3.2 Limitations
346
14 Git
347
14.1 Git Crash Course
348
14.1.1 Preparation Tasks
348
14.1.2 The First Repository
349
14.1.3 The First Commit
350
14.1.4 Additional Commits
351
14.1.5 Setting Up the Repository on a Second Computer
351
14.1.6 Git Status
353
14.1.7 Excluding Files from Git (.gitignore)
353
14.1.8 Transferring Existing Code to a New Repository
354
14.2 Handling Settings and Passwords Correctly
355
14.2.1 What’s a Better Method?
356
14.2.2 Example
357
14.3 Git Automation
358
14.3.1 Controlling GitHub Remotely Using gh
360
14.4 Git Hooks
360
14.4.1 Example: Detecting Unversioned Files Before the Commit
361
PART III Applications and Examples
363
15 Backups
365
15.1 Synchronizing Directories to External Storage
365
15.1.1 PowerShell Script with robocopy
366
15.1.2 Ideas for Improvement
368
15.1.3 Bash Script with rsync
368
15.2 WordPress Backup
370
15.3 SQL Server Backup
372
15.3.1 Backing Up and Compressing all Databases via Script
373
16 Image Processing
375
16.1 Manipulating Image Files
375
16.1.1 Installing ImageMagick
375
16.1.2 Trying Out ImageMagick
376
16.1.3 Example: convert2eps (Bash Variant)
378
16.1.4 Example: convert2eps (PowerShell Variant)
379
16.2 Sorting Photos by Date Taken
379
16.2.1 Installing and Trying Out ExifTool
380
16.2.2 Bash Example
381
16.2.3 PowerShell Example
382
16.3 Converting Exif Metadata to SQL Commands
382
16.3.1 PyExifTool
383
16.3.2 EXIF2SQL
384
16.3.3 Options for Enhancement
387
17 Web Scraping
389
17.1 Limitations
389
17.2 Web Scraping, Web Crawling, and Data Mining
390
17.3 Downloading Websites Using wget
390
17.3.1 Example 1: Downloading Directly Linked Images
391
17.3.2 Example 2: Downloading all PDF Files Linked on the Website
392
17.3.3 Example 3: Creating a Local Copy of the Website
392
17.4 Web Scraping with Regular Expressions
393
17.5 Web Scraping with Beautiful Soup
395
17.5.1 Beautiful Soup and Requests
395
17.5.2 Hello, Beautiful Soup!
395
17.5.3 Example: Determining a List of Top Titles from Rheinwerk Publishing
396
17.6 Web Scraping with Requests-HTML
399
17.6.1 Example: Analyzing the GitHub Status
399
17.7 Web Scraping with PowerShell
401
17.7.1 PowerHTML Module
403
18 Using REST APIs
405
18.1 Tools
405
18.2 Sample APIs to Try Out
406
18.3 Implementing Custom REST APIs
406
18.4 curl and wget
406
18.4.1 curl
407
18.4.2 wget
409
18.4.3 Using REST APIs in Bash Scripts
410
18.5 Using REST APIs in PowerShell
411
18.5.1 Options
412
18.6 Example: Determining the Current Weather
413
18.7 Using REST APIs in Python
414
18.8 Example: Determining Electricity Prices and Displaying Them Graphically
416
18.8.1 aWATTar API
417
18.8.2 Analysis of the Data
417
18.8.3 Matplotlib
419
18.8.4 Controlling the Energy Consumption
419
19 Databases
421
19.1 Updating and Maintaining Databases
422
19.1.1 PowerShell and sqlcmd
422
19.2 Creating a New Customer Account
423
19.2.1 Account Data
424
19.2.2 Structure of the Script
424
19.3 Storing Exif Metadata in a Database
426
19.3.1 PyMySQL
427
19.3.2 Saving Exif Metadata
428
19.4 Importing JSON Data into a Table
429
20 Scripting in the Cloud
433
20.1 AWS CLI
433
20.1.1 Installation on Linux and macOS
434
20.1.2 Configuration
434
20.1.3 Getting Started
435
20.1.4 Encrypting Files
436
20.2 Example: Uploading Encrypted Backup Files to the Cloud
438
20.3 AWS PowerShell Module
439
20.3.1 Getting Started
439
20.3.2 Copying Files
441
20.4 Example: Offloading Large Files from a Website to the Cloud
442
20.4.1 Preparations
443
20.4.2 Script
443
20.4.3 Limitations
445
21 Virtual Machines
447
21.1 Setting Up and Running Virtual Machines (KVMs)
447
21.1.1 Cloning Virtual Machines
448
21.1.2 Starting and Shutting Down Virtual Machines
449
21.1.3 Running Scripts on Multiple Virtual Machines
449
21.2 Automating the Network Configuration (KVMs)
450
21.2.1 Starting Point
451
21.3 Controlling Hyper-V
453
21.3.1 Cloning Virtual Machines
455
The Author
459
Index
461