#38:05 is when he comes in re: 2.3

[aq2003]

I'm late to the game but, similar to much ado about nothing I'm intrigued by this twelfth night business! I'm not sure where to listen to the version you've been talking about though. All I can find are film versions.

you can listen to it here! and you can find an annotated ebook version of the play to read alongside it here. it's REALLY fucking good, probably one of my favorite shakespeare adaptations and not just because david tennant gave me malvolio derangements in it. it's just really excellently done - i feel like you can so easily imagine the setting and the characters etc SO well from the performances and the music

they split up and moved around the acts/scenes so i'll timestamp them for clarity here but imo the choices they made were all so effective! (for instance: starting with actually showing the shipwreck and viola and sebastian getting separated, then jumping right into act 1 scene 2 with viola and the captain. then act 1 scene 1 comes afterwards, with viola/cesario already being in orsino's employ. they're the one playing the music for him in the scene and they take up curio's lines. it sets up viola and orsino's dynamic all the more esp with orsino's first line being "if music be the food of love" - such a great adaptation decision!)

act 1 scene 2: 0:00-5:05

act 1 scene 1: 5:06-7:45

act 1 scene 4 (pt 1): 7:46-8:19

act 1 scene 3: 8:20-14:03

act 1 scene 4 (pt 2): 14:04-16:10

act 1 scene 5: 16:11-29:20

-

act 2 scene 2: 29:21-32:15

act 2 scene 3: 32:16-42:50

act 2 scene 4: 42:51-48:30

act 2 scene 1: 48:31-51:12

act 2 scene 5: 51:13-1:02:42

-

act 3 scene 1: 1:02:43-1:08:38

act 3 scene 2: 1:08:39-1:12:07

act 3 scene 3: 1:12:08-1:14:29

act 3 scene 4: 1:14:30-1:32:00

-

act 4 scene 1: 1:32:01-1:35:01

act 4 scene 2 (pt 1): 1:35:02-1:37:53

act 4 scene 3 (pt 1): 1:37:54-1:39:13

act 4 scene 2 (pt 2): 1:39:14-1:41:24

act 4 scene 3 (pt 2): 1:41:25-1:42:34

-

act 5 scene 1: 1:42:35-2:01:23

[theresawelchy]

Systems Monitoring: top vs Htop vs Glances

When developing a piece of software or monitoring a running system both telemetry and context are important. After one understands what is normal behaviour in a historical context the two most pressing questions are often (1) what's changed? and (2) what's acting abnormally?

In this post I'm going to look at three popular tools often used for ad-hoc monitoring as well as look at a simplistic solution for monitoring distributed systems.

top

In virtually any modern UNIX-like system you can type top and see a variety of system performance metrics updating every few seconds.

top - 09:43:05 up 1:08, 0 users, load average: 0.52, 0.58, 0.59 Tasks: 4 total, 1 running, 3 sleeping, 0 stopped, 0 zombie %Cpu0 : 4.1 us, 22.2 sy, 0.0 ni, 72.3 id, 0.0 wa, 1.4 hi, 0.0 si, 0.0 st %Cpu1 : 4.3 us, 7.1 sy, 0.0 ni, 87.7 id, 0.0 wa, 0.9 hi, 0.0 si, 0.0 st %Cpu2 : 4.4 us, 9.0 sy, 0.0 ni, 85.3 id, 0.0 wa, 1.2 hi, 0.0 si, 0.0 st %Cpu3 : 3.6 us, 6.7 sy, 0.0 ni, 88.6 id, 0.0 wa, 1.0 hi, 0.0 si, 0.0 st KiB Mem: 33431016 total, 9521052 used, 23909964 free, 34032 buffers KiB Swap: 62455548 total, 27064 used, 62428484 free. 188576 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 8304 132 104 S 0.0 0.0 0:00.14 /init ro 3 root 20 0 8308 96 56 S 0.0 0.0 0:00.00 /init ro 4 mark 20 0 17856 5308 5192 S 0.0 0.0 0:00.35 -bash 228 mark 20 0 14452 1668 1172 R 0.0 0.0 0:00.01 top -b -n2 -d5

The binary running is most like a version of top written by James Warner of Comcast. This version of top is entirely new and was built as a replacement to a previous version written by developers from a variety of organisations including Lockheed Martin and Heidelberg University.

The top.c source code itself is reasonably simplistic and as of this writing was around ~4,900 lines of C code. Top is still in active development to this day and its source code can be seen with the rest of the procps repository on GitLab. Other utilities found in this repo include kill, ps, sysctl, uptime and watch.

The default layout feels timeless to me but over the decades I've been working with UNIX systems I've developed a mussel memory for typing zc1M every time I bring up top on a new machine.

Top uses a monochrome display mode by default so z will toggle into a colour-mapping mode. The number 1 will display separate CPU states and does a good job at highlighting single CPU core-bound loads. I like to view processes sorted by their pressure on memory capacity by typing M. In total there are 49 metrics top can view and sort on.

Commands are truncated by default and typing c will give more extended information on their paths and arguments. My only complaint with this is that it's the end of the commands and arguments that are truncated; it would be more useful to just keep the beginning and end of each command and argument string in order to differentiate between processes.

The changes to top's configuration will only last as long as the session. To avoid this, type uppercase W and it'll save the current configuration to ~/.toprc by default. My only annoyance with this file is that it contains byte values above 0x7F and isn't easy to edit outside of top.

$ hexdump -C ~/.toprc | head

00000000 74 6f 70 27 73 20 43 6f 6e 66 69 67 20 46 69 6c |top's Config Fil| 00000010 65 20 28 4c 69 6e 75 78 20 70 72 6f 63 65 73 73 |e (Linux process| 00000020 65 73 20 77 69 74 68 20 77 69 6e 64 6f 77 73 29 |es with windows)| 00000030 0a 49 64 3a 69 2c 20 4d 6f 64 65 5f 61 6c 74 73 |.Id:i, Mode_alts| 00000040 63 72 3d 30 2c 20 4d 6f 64 65 5f 69 72 69 78 70 |cr=0, Mode_irixp| 00000050 73 3d 31 2c 20 44 65 6c 61 79 5f 74 69 6d 65 3d |s=1, Delay_time=| 00000060 33 2e 30 2c 20 43 75 72 77 69 6e 3d 30 0a 44 65 |3.0, Curwin=0.De| 00000070 66 09 66 69 65 6c 64 73 63 75 72 3d a5 a8 b3 b4 |f.fieldscur=....| 00000080 bb bd c0 c4 b7 ba b9 c5 26 27 29 2a 2b 2c 2d 2e |........&')*+,-.| 00000090 2f 30 31 32 35 36 38 3c 3e 3f 41 42 43 46 47 48 |/012568<>?ABCFGH|

Htop

In 2004, Hisham Muhammad began work on creating a distinctly different systems telemetry monitor. Htop put a focus on telemetry display organisation. There are bar charts for key CPU and memory metrics, processes can toggle between a flat list and a hierarchy via the F5 shortcut, field sorted can be done via mouse clicks and there are seven different colour modes are supported.

The software does a good job of keeping you within the application. If you want to inspect the files a process is using you can select the process and simply type l, if you want to run the process through strace simply type s while running htop as a privileged user.

Below will install and run htop on Ubuntu 16.04.2 LTS.

$ sudo apt install htop $ htop

1 [ 0.0%] Tasks: 37, 145 thr; 1 running 2 [ 0.0%] Load average: 0.03 0.05 0.07 3 [ 0.0%] Uptime: 01:31:42 4 [ 0.0%] Mem[|||||||||||||||||||||||||||||||| 1.03G/3.84G] Swp[ 0K/4.00G] PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 1 root 20 0 37556 5668 4004 S 0.0 0.1 0:03.03 /sbin/init noprompt 27884 clickhous 20 0 3716M 359M 49184 S 0.7 9.1 0:24.93 ├─ /usr/bin/clickhouse-server --config=/etc/cli 29668 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:00.10 │ ├─ /usr/bin/clickhouse-server --config=/etc/ 29667 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:01.02 │ ├─ /usr/bin/clickhouse-server --config=/etc/ 29666 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:00.08 │ ├─ /usr/bin/clickhouse-server --config=/etc/ 29665 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:00.48 │ ├─ /usr/bin/clickhouse-server --config=/etc/ 29409 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:03.48 │ ├─ /usr/bin/clickhouse-server --config=/etc/ 29408 clickhous 20 0 3716M 359M 49184 S 0.0 9.1 0:02.15 │ ├─ /usr/bin/clickhouse-server --config=/etc/

In terms of configuration, any changes made while using the software will be saved automatically to ~/.config/htop/htoprc by default. This file is text-based but comes with the following warning:

$ head -n2 ~/.config/htop/htoprc

# Beware! This file is rewritten by htop when settings are changed in the interface. # The parser is also very primitive, and not human-friendly.

The source code is still quiet small given the functionality on offer. As of this writing there's a total of ~12,000 lines of C code with other files making up a further ~3,000 lines of code.

Glances

Glances is a Python-based systems telemetry monitor. The project was started by Nicolas Hennion in 2011. Nicolas' LinkedIn profile states he works in the South of France as a Project Manager in the Satellite Control Centre Department for Thales Alenia Space.

When you launch Glances, in addition to the regular CPU, memory and process lists, you'll see the Cloud virtual machine type as well as network, disk and docker container activity to name just a few items.

ubuntu (Ubuntu 16.04 64bit / Linux 4.4.0-62-generic) Uptime: 18:55:00 CPU [ 1.7%] CPU - 1.7% nice: 0.0% ctx_sw: 923 MEM - 53.1% SWAP - 0.1% LOAD 4-core MEM [ 53.1%] user: 0.8% irq: 0.0% inter: 587 total: 3.84G total: 4.00G 1 min: 0.20 SWAP [ 0.1%] system: 0.7% iowait: 0.0% sw_int: 786 used: 2.04G used: 3.27M 5 min: 0.14 idle: 98.4% steal: 0.0% free: 1.80G free: 3.99G 15 min: 0.10 NETWORK Rx/s Tx/s TASKS 203 (349 thr), 1 run, 202 slp, 0 oth sorted automatically by CPU consumption ens33 152b 3Kb lo 59Kb 59Kb CPU% MEM% VIRT RES PID USER TIME+ THR NI S R/s W/s Command 2.6 4.5 524M 178M 16470 mark 35:48 1 0 S 0 0 /home/mark/. DISK I/O R/s W/s 2.3 0.6 372M 24.5M 14672 mark 0:01 1 0 R 0 0 /home/mark/. fd0 0 0 1.0 23.7 5.42G 931M 21151 root 13:00 71 0 S ? ? java -Xmx1G loop0 0 0 0.7 9.8 3.71G 385M 27884 clickhous 5:29 46 0 S ? ? /usr/bin/cli loop1 0 0 0.3 2.8 3.53G 109M 12883 zookeeper 1:36 20 0 S ? ? /usr/bin/jav loop2 0 0 0.3 0.2 31.4M 6.80M 333 root 0:53 1 0 S ? ? /lib/systemd loop3 0 0 0.3 0.1 13.8M 2.68M 4353 mark 1:07 1 0 S 0 0 watch ifconf loop4 0 0 0.0 0.3 186M 9.86M 1447 root 0:35 2 0 S ? ? /usr/bin/vmt loop5 0 0 0.0 0.2 75.2M 8.11M 1470 root 0:00 1 0 S ? ? /usr/bin/VGA loop6 0 0 0.0 0.2 90.6M 6.59M 4381 root 0:00 1 0 S ? ? sshd: mark [ loop7 0 0 0.0 0.1 269M 5.75M 595 root 0:13 3 0 S ? ? /usr/lib/acc sda 0 78K 0.0 0.1 36.7M 5.37M 1 root 0:37 1 0 S ? ? /sbin/init n sda1 0 78K 0.0 0.1 64.0M 5.31M 4246 root 0:00 1 0 S ? ? /usr/sbin/ss sda2 0 0 0.0 0.1 44.3M 5.05M 3402 mark 0:00 1 0 S 0 0 /lib/systemd sda5 0 0 0.0 0.1 21.8M 5.04M 4403 mark 27:23 1 0 S 0 0 -bash sr0 0 0 0.0 0.1 21.8M 4.93M 21493 mark 0:10 1 0 S 0 0 /bin/bash sr1 0 0 0.0 0.1 21.7M 4.62M 16114 mark 0:03 1 0 S 0 0 /bin/bash 0.0 0.1 21.7M 4.47M 21119 mark 0:00 1 0 S 0 0 /bin/bash FILE SYS Used Total 0.0 0.1 90.6M 4.14M 4402 mark 0:08 1 0 S ? ? 0 / (sda1) 2.48G 15.6G 0.0 0.1 250M 3.97M 588 syslog 0:28 4 0 S ? ? /usr/sbin/rs 0.0 0.1 21.8M 3.87M 3407 mark 0:04 1 0 S 0 0 -bash SENSORS 0.0 0.1 51.5M 3.76M 21144 root 0:00 1 0 S ? ? sudo nohup / Physical id 100C 0.0 0.1 41.9M 3.64M 597 messagebu 0:00 1 0 S ? ? /usr/bin/dbu Core 0 100C 0.0 0.1 43.2M 3.45M 396 root 0:01 1 0 S ? ? /lib/systemd Core 1 100C 0.0 0.1 64.3M 3.21M 3377 root 0:00 1 0 S ? ? /bin/login - Core 2 100C 0.0 0.1 28.0M 2.91M 592 root 0:00 1 0 S ? ? /lib/systemd Core 3 100C 0.0 0.1 26.7M 2.86M 16113 mark 0:06 1 0 S ? ? SCREEN 0.0 0.1 15.7M 2.81M 774 root 0:00 1 0 S ? ? /sbin/dhclie

Glances is written with ~10K lines of Python, ~25K lines of JavaScript and relies on the psutil package for it's telemetry collection. There are a huge variety of plugins including support for monitoring GPUs, Kafka, RAID setups, folder monitoring and WiFi to name a few.

In addition to the ncurses-based interface, Glances can also run as a web application. When you run glances via cmd.exe on Windows 10 it'll launch a Bottle-based Web Application on TCP port 61209. When you load up http://127.0.0.1:61209/ in a web browser you'll be greeted with an AngularJS-based Application that mimics the ncurses interface.

There is an API exposed as well if you want to consume it with other tools.

$ curl http://127.0.0.1:61209/api/3/all \ | python -mjson.tool \ | head -n50

{ "alert": [], "amps": [], "batpercent": [], "cloud": {}, "core": { "log": 4, "phys": 4 }, "cpu": { "cpucore": 4, "ctx_switches": 182358, "idle": 82.9, "interrupts": 113134, "soft_interrupts": 0, "syscalls": 215848, "system": 12.5, "time_since_update": 8.532670974731445, "total": 9.8, "user": 3.1 }, "diskio": [ { "disk_name": "PhysicalDrive6", "key": "disk_name", "read_bytes": 0, "read_count": 0, "time_since_update": 8.492774963378906, "write_bytes": 0, "write_count": 0 }, { "disk_name": "PhysicalDrive2", "key": "disk_name", "read_bytes": 0, "read_count": 0, "time_since_update": 8.492774963378906, "write_bytes": 0, "write_count": 0 }, ...

The default configuration file is somewhat lengthy but is friendly enough for human editing.

Glances also supports exporting telemetry to over 16 different targets including statsd, Kafka, RabbitMQ, JSON, SVG, Elasticsearch, CSV as well as to bespoke RESTful APIs.

Feeding Glances into Kafka

Below I'll walk through exporting telemetry to a CSV file and then feeding that into Kafka. My thinking behind this is that local disk is usually more reliable than network connections and if the network connection were to fail the local file could be backfilled into Kafka again.

The following was run on a fresh installation of Ubuntu 16.04.2 LTS.

$ sudo apt update $ sudo apt install \ kafkacat \ python-pip \ python-virtualenv \ screen \ zookeeperd

I'll install Kafka manually using the binary package distributed by one of Apache's mirrors.

$ sudo mkdir -p /opt/kafka $ wget -c -O kafka.tgz \ http://www-eu.apache.org/dist/kafka/1.1.1/kafka_2.11-1.1.1.tgz $ sudo tar xzvf kafka.tgz \ --directory=/opt/kafka \ --strip 1

I'll then create a log file for Kafka which will be owned by my UNIX account.

$ sudo touch /var/log/kafka.log $ sudo chown mark /var/log/kafka.log

Much of Kafka's distributed functionality is supported by ZooKeeper. The following command will launch the service.

$ sudo /etc/init.d/zookeeper start

With ZooKeeper up I'll launch Kafka's server process.

$ sudo nohup /opt/kafka/bin/kafka-server-start.sh \ /opt/kafka/config/server.properties \ > /var/log/kafka.log 2>&1 &

I'll then create a Python virtual environment and install Glances as well as CSVKit so I can analyse the CSV output from Glances.

$ virtualenv ~/.monitoring $ source ~/.monitoring/bin/activate $ pip install \ csvkit \ glances

Below I'll launch a screen session and start Glances. It will both display the ncurses interface as well as write 215 pieces of telemetry to ~/glances.csv.

$ screen $ glances --export csv \ --export-csv-file ~/glances.csv

Once that's running type CTRL-A and then CTRL-D to return to your regular shell.

As you can see, there is a large amount of telemetry being collected.

$ csvstat --type ~/glances.csv | tail

206. mem_available: Number 207. mem_used: Number 208. mem_cached: Number 209. mem_percent: Number 210. mem_free: Number 211. mem_inactive: Number 212. mem_active: Number 213. mem_shared: Number 214. mem_total: Number 215. mem_buffers: Number

Kafkacat is a non-JVM Kafka producer and consumer that's written in C. When statically linked it's less than 150 KB in size. Below I'll use it to feed the contents of ~/glances.csv into a Kafka topic called "glances_log" and use Snappy compression on the contents.

$ screen $ tail -F ~/glances.csv \ | kafkacat -b localhost:9092 \ -t glances_log \ -z snappy

Again, once that's running type CTRL-A and then CTRL-D to return to your regular shell.

Any of the above commands running in screen sessions could be easily added to Supervisord. It would do a good job of restarting processes if they were to fail for any reason.

With all the above running I'll take a look at the first three columns of data for the first 100 records.

$ /opt/kafka/bin/kafka-console-consumer.sh \ --topic glances_log \ --from-beginning \ --zookeeper localhost:2181 \ | head -n100 \ | csvstat --columns 1-3 \ --no-header-row

Below are the statistics for the timestamp column, number of CPU cores and 1-minute load averages collected in the first 100 records.

1. "a" Type of data: DateTime Contains null values: False Unique values: 100 Smallest value: 2018-10-07 05:53:49 Largest value: 2018-10-07 05:58:55 Most common values: 2018-10-07 05:53:49 (1x) 2018-10-07 05:53:52 (1x) 2018-10-07 05:53:55 (1x) 2018-10-07 05:53:58 (1x) 2018-10-07 05:54:01 (1x) 2. "b" Type of data: Number Contains null values: False Unique values: 1 Smallest value: 4 Largest value: 4 Sum: 400 Mean: 4 Median: 4 StDev: 0 Most common values: 4 (100x) 3. "c" Type of data: Number Contains null values: False Unique values: 18 Smallest value: 0.02 Largest value: 0.22 Sum: 6.57 Mean: 0.066 Median: 0.05 StDev: 0.045 Most common values: 0.04 (15x) 0.02 (14x) 0.03 (13x) 0.06 (9x) 0.05 (9x)

Thank you for taking the time to read this post. I offer consulting, architecture and hands-on development services to clients in North America & Europe. If you'd like to discuss how my offerings can help your business please contact me via

LinkedIn

.

DataTau published first on DataTau