Tagged: Linux

‘Linux Command-Line Tips & Tricks’ paperback and ebook updated

Lots of new tips and tricks added. Code snippets are syntax highlighted (only in paperback).

Last week, I moved the last remaining of my books from Ubuntu 10.10 to Linux Mint 20. I changed some covers. I developed a Javascript trick to detect missing images in HTML. I also learned to use ImageMagick to detect images with CMYK profiles and less than 300 DPI. Book cover images in my backlist had CMYK profiles. These came from the KDP cover template and khtmltopdf would complain about them. I also wrote two articles for Open Source For You magazine.

  • How to tame Mozilla Firefox
    This article was about how to lock down Firefox using policies.json file.
  • BASH Secrets Nobody Talks About
    The other article was about some new and important things that I discovered about bash shell programming language.

All of these discoveries were added to my book Linux Command-Line Tips & Tricks. My book-creation process is highly automated. Highly, not fully. The paperback (PDF) creation process is fully automated with shell scripts. My ebook creation process is still partly manual. Calibre has a command-line interface so in future I will fully automate that too.

Book cover of Linux Command-Line Tips & Tricks

How to composite videos using FFMPEG

Video overlays are not easy but can be done.

I have two videos and I need to place them side-by-side in a new video. The dimensions first video (blob.mp4) was 1280×720. The second video (aoc.mp4) was 406×720.

The compositing was achieved by using a complex filter:

ffmpeg -i blob.mp4 -i aoc.mp4 -filter_complex "[0:v]pad=1686:720:0:[a]; [a][1:v]overlay=1280:0[b]" -map "[b]" -pix_fmt yuv420p aoc-meets-garbage-disposal.mp4

In the first part of the filter, I specified that the first input video (blob.mp4) will have a dimension of 1686×720, where 1686 is the total width of the both videos side-by-side. A map identifier “[a]” is used for this screen canvas. Next, with a semicolon delimiter, the second video is specified as appearing at 1280 pixels from the left edge at 0. (Yes, it begins at zero. I got a stupid “Overlay area … not within the main area … or zero-sized… Failed to configure input pad on Parsed_overlay_1” error when I thought I had to use 1281 as the starting point of the second video.) The overlaid video is then map identified as “[b]”. This map is then encoded to the output video.

Of course, this video does not have audio. So, I extracted the audio from both videos using FFMPEG, mixed them using Audacity (although I could have done it using FFMPEG as well), and slipstreamed the audio to the composite video using FFMPEG.

ffmpeg -i blob.mp4 -vn blob.mp3
ffmpeg -i aoc.mp4 -vn aoc.mp3

#Created garbage.mp3 from blob.mp3 and aoc.mp3 using Audacity

ffmpeg -i aoc-meets-garbage-disposal.mp4 -i garbage.mp3 -codec copy aoc-meets-garbage-disposal-mix.mp4

Alexandria Ocasio-Cortez, the US Congresswoman, lived in New York where garbage disposal machines were banned and people did not buy them even after the ban was lifted. She recently caused worldwide hilarity after posting an ominous video about her discovery of the garbage disposal machine in her sink. She must have at least seen them in movies or TV.

Subhash TweetsToRSS v2019.03.04 for Android, Linux, Mac and Windows

Mostly bug fixes.

  • The search form at the top of results page has been revamped and made more meaningful. You can specify the number of results per search.
  • The bug which causes the “older tweets” link in the footer to omit the location information has been fixed.

Search form of Subhash TweetsToRSS

Subhash TweetsToRSS search form is more meaningful.

JAR executable (Linux/Mac/Windows) and APK installer (for Android) available at:
http://www.vsubhash.com/subhash-tweetstorss-app-for-android-linux-mac-and-windows.html

EmailTweetor (for Desktop) – the command-line Twitter client released for Linux, Mac and Windows

Like the Android version, it can tweet, reply, delete, favorite and retweet but it cannot alas email.

This terminal-based Twitter client can be operated interactively or automated using command-line parameters. To prevent your bash history from getting cluttered with unsightly tweets, you cannot specify the message in the command line. However, you can do pipe redirection or use a text file. The Twitter client is extremely powerful so I have not documented all of its features in the help.

EmailTweetor was made with Oracle Java version 1.8.0_101. So, ensure that your JVM (Oracle or OpenJDK) is newer or equivalent.

EmailTweetor lets you tweet from the command line.

http://www.vsubhash.com/subhash-email-tweetor-app-for-android.html

Compile from source and install Apache HTTP Server 2.4 and PHP 7 on Linux

A how-to guide to build localhost (“virtual host”) PHP 7 sites on the latest Apache HTTP server

I am using a nearly decade-old version of Ubuntu as my main OS. As it runs Gnome 2 and Firestarter, I will be sticking with it until rapture time. However, I did not want the old Apache and PHP from the “Old Releases” Ubuntu repository. Apache HTTP Server 2.4 was released in March this year. I couldn’t care for the new features. The old Apache would have worked just fine but new was new. So, I decided to install the latest Apache HTTP server and PHP 7 by building them from source. There aren’t any tutorials on building Apache 2.4 and PHP 7 from source, particularly from the requirement of a development machine. Besides that many of the configuration files, executables (a2ensite) and directories seem to be missing after the compilation/installation process was complete. It required a bit of thinking and RTFM. So, I decided to put it all down here in one place.

Apache HTTP Server 2.4 (httpd) compilation, installation and configuration

First, download the source files.

cd ~/Build
wget -c url-of-pcre-archive
wget -c url-of-apache-httpd-archive 
wget -c url-of-apr-util
wget -c url-of-apr

Extract the source files of PCRE and httpd in a directory named Build. The source files of the Apache Runtime need to be extracted inside the srclib directory of the httpd directory.

cd ~/Build
tar -xf pcre*
tar -xjf http*.bz*

cd http*
cd srclib
tar -xf ../../apr-1*
tar -xf ../../apr-util-1*
ln -s apr-1* apr
ln -s apr-util-1* apr-util

Next, compile and install PCRE to some directory (/usr/local/pcre).

cd ~/Build
cd pcre*
./configure --prefix=/usr/local/pcre
make
sudo make install # Installs PCRE to directory /usr/local/pcre

Finally, compile Apache httpd to some directory (/opt/httpd) with specified PCRE and Apache Runtime directories.

cd ~/Build
cd http*
./configure --prefix=/opt/httpd --with-included-apr --with-pcre=/usr/local/pcre
make
sudo make install # Install Apache httpd to directory /opt/httpd

The Apache HTTP Server has been installed. Now, it is time to start it and check it in a browser.

sudo /opt/httpd/bin/apachectl -k start # Start server
firefox http://localhost  # Serves /opt/httpd/htdocs/index.html

The configuration files of Apache httpd 2.4 seem to be different from the ones from the Apache installed from the old Ubuntu repositories. The main configuration file will be at /opt/httpd/conf/httpd.conf

PHP7 compilation, installation and configuration

Next, compile and install PHP 7 to some directory (/opt/php-7). When executing the configure statement, you need to specify the location of the Apache Extension Tool (/opt/httpd/bin/apxs).

sudo /opt/httpd/bin/apachectl -k stop # Stop Apache HTTP Server
cd ~/Build
cd php-7*
./configure --help | more
read -p "Press enter to continue" oNothing 
./configure -with-apxs2=/opt/httpd/bin/apxs  --prefix=/opt/php/php-7/arch-all --exec-prefix=/opt/php/php-7/arch
make
make test # This step is optional 
sudo make install # Install PHP to directory /opt/php/php-7
sudo libtool --finish ~/Build/php-7/libs # Adds PHP 7 module loader to httpd.conf

This completes the PHP 7 installation. Please note that the libtool statement writes the following line to Apache server’s httpd.conf file.

LoadModule php7_module        modules/libphp7.so

After PHP 7 has been installed, add a PHP file extension handler to Apache server’s httpd.conf file.

# The following is a multi-line statement. Copy all of them.
echo '
  <FilesMatch \.php>; 
    SetHandler application/x-httpd-php 
  </FilesMatch>' | sudo tee -a /opt/httpd/conf/httpd.conf 

Copy the development php.ini in the Build directory to /opt/php/php-7/arch/lib directory.

cd ~/Build/php-7*
sudo cp php.ini-development /opt/php/php-7/arch/lib/php.ini

Modify this file appropriately for your developmental setup. For production, you will have to use modify and use php.ini-production as the php.ini file. This completes the PHP installation and configuration.

Now, it is time to test .it. Create a sample PHP file in the Apache server’s htdocs directory and request it in a browser.

echo '<? phpinfo(); ?>' > php-test.php
sudo mv ./php-test.php /opt/httpd/htdocs/php-test.php
sudo /opt/httpd/bin/apachectl -k start # Start Apache HTTP Server
firefox http://localhost/php-test.php

This will prove that your HTTPd server can handle PHP. However, to develop websites on your local development machine, some more changes need to be made. You can delete the /opt/httpd/htdocs/php-test.php file.

Apache HTTP Server reconfiguration for virtual hosts

First, append a line in your /etc/hosts file so that browser requests to your test site (say me-website.com) are routed to your local installation of the httpd server.

echo "127.0.0.1 me-website.com" | sudo tee -a /etc/hosts

Next, clean up httpd.conf by removing the PHP file handler and the PHP loadmodule statement. Instead, append the following lines to httpd.conf.

# Added by me
Include conf/me-websites.conf 

Now, httpd.conf is mostly in its original state, save for the line that tells httpd to load the new me-websites.conf configuration file. Then, create the file me-websites.conf file and put all your virtual hosts configurations in it. Copy this file to /opt/httpd/conf directory. This will ensure that PHP scripting is limited to your “virtual host” sites.

Listen 8080

<VirtualHost *:8080>
  ServerAdmin webmaster@me-website.com
  DocumentRoot "/home/account-name/Websites/PHP/me-website/htdocs"
  ServerName me-website.com
  ServerAlias www.me-website.com
  ErrorLog "/home/account-name/Websites/PHP/me-website/logs/me-website.com.error.log"
  TransferLog "/home/account-name/Websites/PHP/me-website/logs/me-website.com.access.log"
  LoadModule rewrite_module modules/mod_rewrite.so
  LoadModule php7_module        modules/libphp7.so

  <FilesMatch \.php$>
    SetHandler application/x-httpd-php
  </FilesMatch>
  
  <Directory "/home/account-name/Websites/PHP/me-website/htdocs">
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>

</VirtualHost>

Next, add your account to the user group www-data. Your account can be the owner of the directory /home/account-name/Websites/PHP/me-website/htdocs but you need to give read, write and execute permissions to www-data. Now, you can put your PHP scripts in /home/account-name/Websites/PHP/me-website/htdocs and start coding. You can test the site in a browser at http://me-website.com:8080/

If you would like to test it on port 80, then remove the “Listen 80” in httpd.conf and replace 8080 in inside the VirtualHost directive in me-websites.conf. Then, you can simply test your sites at http://me-website.com/ or http://me-website.com:80.

Fix for autogenerate bug in Eclipse PHP run configuration

I have already published how to modify the Eclipse source files to fix the autogenerate bug in the Run configuration of Eclipse PHP. If that is too much for you, then you can simply let Apache HTTP Server’s rewrite module fix it for you. Create a .htaccess file in your home folder. This will remove the your Eclipse PHP project name for the run configuration.

RewriteEngine On
Redirect "/me-website" "/"

Interestingly, the propriety IDE (derived from the same open-source project) sold by the creator of PHP does not suffer from this PDT “bug”.

Create a Twitter Archive – A PDF dump of all tweets from an account

I have 51 tweets in my @SubhashBrowser account. So, this command will create the Twitter dump.

bash twitter-to-pdf.txt SubhashBrowser 51

Subhash TweetsToRSS must be running when you execute this command. You also need wkhtmltopdf and pdftk installed. The content of twitter-to-pdf.txt is as follows:

# Initialize variables
iPage=1
iMax=1
iRemainder=0
sDocs=""

let "iMax = $2 / 25"
let "iRemainder = $2 % 25"
if [ $iRemainder -gt 0 ]; then
  let "iMax = iMax + 1"
fi

sJavaScript = 
	"(function(){ " + 
	"  var arIframes=document.getElementsByTagName('iframe');" +
	"  var n = arIframes.length; " + 
	"  for (var i=0;i<n;i++) { " + 
	"  arIframes[0].parentElement.removeChild(arIframes[0]); " +
	"}})()" # Javascript to remove YouTube iframes generated by TweetsToRSS

# Create archives
for (( iPage=1; iPage<=iMax; iPage++ ))
do
  echo "Converting http://localhost:8080/?q=%40$1&output=html&older-than=$iPage"
  wkhtmltopdf --quiet \
              --encoding utf-8 \
              --debug-javascript --javascript-delay 60 --run-script "$sJavaScript" 	\
              --user-style-sheet wkhtml_twitter_style.css \
              --image-dpi 300 -s A3 \
              --margin-top 0 --margin-right 0 --margin-bottom 0 --margin-left 0 \
              "http://localhost:8080/?q=%40$1&output=html&older-than=$iPage" \
              "@$1-archive-$iPage.pdf"
  sDocs="$sDocs @$1-archive-$iPage.pdf"	 					  
done

# Combine archives
pdftk $sDocs cat output Complete-Twitter-Archive-of-@$1.pdf

# Updated metadata
echo "InfoKey: Creator" > $1-meta.txt
echo "InfoValue: Subhash TweetsToRSS (www.vsubhash.com)" >> $1-meta.txt
echo "InfoKey: Title" >> $1-meta.txt
echo "InfoValue: The complete Twitter archive of @$1" >> $1-meta.txt
echo "InfoKey: Subject" >> $1-meta.txt
echo "InfoValue: Collection of all tweets from @$1" >> $1-meta.txt
echo "InfoKey: Author" >> $1-meta.txt
echo "InfoValue: V. Subhash" >> $1-meta.txt
echo "InfoKey: Keywords" >> $1-meta.txt
echo "InfoValue: $1, twitter, tweets, archive" >> $1-meta.txt
pdftk Complete-Twitter-Archive-of-@$1.pdf \
      update_info $1-meta.txt \
      output The-Complete-Twitter-Archive-of-@$1.pdf

# Cleanup
mkdir $1-backup
if [ -d $1-backup  ]; then
	mv @$1-archive-*.pdf ./$1-backup
	mv Complete-Twitter-Archive-of-@$1.pdf ./$1-backup
	mv $1-meta.txt ./$1-backup
fi

All 51 tweets from my @SubhashBrowser has been dumped into a PDF archive.

All 51 tweets from my @SubhashBrowser has been dumped into a PDF archive.

Twitter seems to have placed a limit of just above 3000 to the number of tweets you can pull from an account. Here is the complete Twitter archive of US President Donald J Trump (@realDonaldTrump).

https://www.scribd.com/document/337137738/The-Complete-Twitter-Archive-of-Donald-J-Trump-RealDonaldTrump-2016-2017

Nimbus theme DEB files for Gnome 2 and Mate desktops