Today I'm going to talk about how I learn a new programming language. Let me start by saying that I've never fully mastered a programming language, or even come close. My roles are that of sysadmin and not of an actual programmer. What generally happens is I'm given some task to accomplish, such as modify an already written report in our home brewed application. I will dive into the report and figure out where I need to start modifying things. Or I need to come up with a new process, but since the rest of the home brewed application is written in PHP, I'll write the new process in PHP as well, just to keep everything consistent.
I learn programming by looking at running code and modifying it to do something else or do the same thing but better or more accurately. I have trouble reading through a book and doing the simple examples and building upon that knowledge. It comes from already knowing a bit about programming and feeling dumb for re-reading what a variable is and how to assign it (pretty much the same in all programming languages).
At my last position I was able to learn a lot of PHP and I became very comfortable with it. I had worked with PHP a bit prior to this, but honestly, I had forgotten most of what I had learned. In this case, that is probably for the best, it was all spaghetti code and crap for the most part. The one thing that helped me the most was a good editor (Netbeans for PHP) and of course documentation.
My new position is with a firm that runs perl for nearly 100% of their code base. I've always had a hard time reading perl code in the past (Mostly nagios plugins). Perl uses a lot of syntax that doesn't explain what its doing just by reading it in english. So my typical take something that works and modify it has been a slower process. I'm able to get the end result that I'm looking for, but I wouldn't say "I UNDERSTAND HOW" this code works or what it honestly does.
This had led me to reading the first few chapters of a book (http://www.perl.org/books/beginning-perl/) so that I may understand the HOW something works and WHY it works, not just that it does. This might be the first programming book I actually read all the way through! So far some of the wizardry of perl is going away because I know or at least know where to look up what "=~" does and not have to guess. Some other resources I've used is of course Stackoverflow.com. Some of the users there leave answers that don't just give an answer, but explain how and why it works and examples of other ways to do the same thing.
How do you guys learn a programming language? What tools do you find are the best for learning a new programming language?
Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts
Thursday, May 10, 2012
Thursday, March 03, 2011
Update to Promiscous Mode on vSphere 4
A few months back I wrote about Enabling Promiscuous Mode on vSphere 4. Well, I have learned some more since then. There isn't much information out there about recording a VoIP stream in a virtual machine environment. I hope to change that today.
First thing is first. If you are using VLANS on your physical network set your Virtual Machine Port Group to be a trunk. Also set the port on the physical switch to be a trunk instead of an access port.

Second thing, if you are using the E1000 NIC driver in your guest OS, turn off VLAN support. The screen shot is from Windows 2008 R2 64-bit, but it is similar in every other Windows OS.

Performance Best Practices for vSphere 4
First thing is first. If you are using VLANS on your physical network set your Virtual Machine Port Group to be a trunk. Also set the port on the physical switch to be a trunk instead of an access port.

Second thing, if you are using the E1000 NIC driver in your guest OS, turn off VLAN support. The screen shot is from Windows 2008 R2 64-bit, but it is similar in every other Windows OS.
Performance Best Practices for vSphere 4
NRPE: Unable to read output and sudo
Thank you to Andrea Leofreddi over at cyberz.org for the blog post Nagios nrpe and sudo: “NRPE: Unable to read output”. This was a tremendous help back when I first started working with my md-raid device and Nagios. I found this entry again while working on my very own plug-in for Nagios, check_supervisorctl.sh.
In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty
For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh
Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}
Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!
In short if you are running either CentOS or RHEL (5+ is all I have tested this with) you need to comment out the line "Defaults requiretty" in the /etc/sudoers file. In order to comment the line out simply add a hash symbol to the beginning of the line like so:
#Defaults requiretty
For the total noob, as I once was:
My command configurations:
command[check_raid]=sudo /usr/local/nagios/libexec/check_md_raid
command[check_supervisorctl]=sudo /usr/local/nagios/libexec/check_supervisorctl.sh
Both of the above lines are on a remote host from the nagios server. The checks are run via NRPE like so:
define service{
use generic-service
host_name
service_description RAID Status
check_command check_nrpe!check_raid
notifications_enabled 1
notification_period 24x7
notification_interval 15
notification_options c,w,u,r
}
define service{
use generic-service
host_name
service_description Supervisor Workers
check_command check_nrpe!check_supervisorctl
notifications_enabled 1
notification_period 24x7
notification_interval 30
}
Without "Defaults requiretty" commented out the output of my sudo command was simply:
NRPE: Unable to read output
But once I disabled requiretty I got the output I expected from my checks:
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H raid.hostname.local -c check_raid
RAID OK: All arrays OK [1 array checked]
[root@hostname ~]# /usr/local/nagios/libexec/check_nrpe -H hostname.local -c check_supervisorctl
OK: All of your programs are running!
Thursday, October 14, 2010
Enabling Promiscuous Mode on vSphere 4
We are in desperate need of a testing server. The testing server needs to be able to listen to the network, through port mirroring, to record VoIP calls. In the physical world this is no problem, but we don't have the budget for another physical box, especially for something as non-critical as testing a program upgrade (even though the program it self is very critical to our business).
A year ago the software vendor said that they don't support virtual environments and their product doesn't work in a virtual environment. During discussions this year however they told the truth that they simply don't know if it will work, and thus "don't support" it. I am calling it lazy. It doesn't take much these days to get a VM host going and configured for testing things out.
Well we have done the leg work for them. I have confirmed that vSphere does allow for promiscuous mode, even from the physical network. To be honest I was a little shocked to see this worked.
Here are the steps to enable promiscuous mode on your vSphere host and the guest VM.
We enable promiscuous mode on the virtual adapter by logging into the vSphere Client, going to the VM Host, clicking on the "Configuration" tab, then on "Networking" in the "Hardware" section on the left. Now click on "Properties..." for the Virtual Switch.

Now enabling promiscuous mode on the vSwitch is pretty simple. Click on the "vSwitch" on the "Ports" tab, as shown below and then click on the "Edit..." button.

Now that we are editing the vSwitch properties click on the "Security" tab. Change the option for Promiscuous mode to "Accept" if it isn't already and hit "OK".

To enable promiscuous mode for the guest we need to break down to the command line. I used putty to SSH into my vSphere host, which I had previously setup. Now you need to edit the .vmx file of the guest that will be listening to the network.
# vi /vmfs/volumes/datastore1/testServer/testServer.vmx
I did a search for ethernet, so the promiscuous mode configuration would be with the rest of the ethernet config. Add the following line to the configuration file:
ethernet0.noPromisc = "FALSE"

And save when you are done. I rebooted my testServer just as a precaution, but I'm not certain it is required.
At this point everything is configured on the VM side of things. Make sure you have port mirroring enabled on your physical switch and give it a test. In my environment I commonly use Wireshark. I did my testing by pinging a server on a mirrored port.
A year ago the software vendor said that they don't support virtual environments and their product doesn't work in a virtual environment. During discussions this year however they told the truth that they simply don't know if it will work, and thus "don't support" it. I am calling it lazy. It doesn't take much these days to get a VM host going and configured for testing things out.
Well we have done the leg work for them. I have confirmed that vSphere does allow for promiscuous mode, even from the physical network. To be honest I was a little shocked to see this worked.
Here are the steps to enable promiscuous mode on your vSphere host and the guest VM.
- Enable promiscuous mode on the virtual adapter.
- Enable promiscuous mode on the vSwitch.
- Enable promiscuous mode for the guest.
- Enable port mirroring on the physical switch (not covered here).
- Test capturing network data.
We enable promiscuous mode on the virtual adapter by logging into the vSphere Client, going to the VM Host, clicking on the "Configuration" tab, then on "Networking" in the "Hardware" section on the left. Now click on "Properties..." for the Virtual Switch.
Now enabling promiscuous mode on the vSwitch is pretty simple. Click on the "vSwitch" on the "Ports" tab, as shown below and then click on the "Edit..." button.
Now that we are editing the vSwitch properties click on the "Security" tab. Change the option for Promiscuous mode to "Accept" if it isn't already and hit "OK".
To enable promiscuous mode for the guest we need to break down to the command line. I used putty to SSH into my vSphere host, which I had previously setup. Now you need to edit the .vmx file of the guest that will be listening to the network.
# vi /vmfs/volumes/datastore1/testServer/testServer.vmx
I did a search for ethernet, so the promiscuous mode configuration would be with the rest of the ethernet config. Add the following line to the configuration file:
ethernet0.noPromisc = "FALSE"
And save when you are done. I rebooted my testServer just as a precaution, but I'm not certain it is required.
At this point everything is configured on the VM side of things. Make sure you have port mirroring enabled on your physical switch and give it a test. In my environment I commonly use Wireshark. I did my testing by pinging a server on a mirrored port.
Using PSExec to Defragment your PCs
Way back in 2000 / 2001 I was an IT intern. My job was to do the really manual processes for a small department with in a much larger company. One of those was monthly defrags of all PCs and Laptops (if they where available). I had to go to each and every PC, log on, and start the defrag process manually. I believe this to be my first moment of "I should automate this". Problem was I didn't know how or really where to start. I knew script was the answer, but just couldn't get things to work. I believe I eventually created a scheduled task on each machine (by hand) to do this for us.
Fast forward 10 years and I am still in need of the same thing, as we don't have Vista or Windows 7 deployed. But now I have a much better understanding of what needs to happen and even better I know where to start!
PSExec, which is part of the System Internals PsTools suite is my answer these days. A simple script, run from my PC (still manually for the time being) is able to handle defragmenting all of our PCs.
psexec @C:\Updates\Comps\AppPCs.txt -n 10 -c -f -d JkDefragCMD.exe
I use the files with PC names to speed up re-deployment of everything. The -n 10 flag tells psexec t0 wait 10 seconds before it times out the PC, instead of 60 (I believe this is the default). -c copies the file (jkdefragcmd.exe) to the remote system. The flag -f forces copying, even if the file exists. The flag -d doesn't wait for the process to terminate, this is as asynchronously as I can do.
My next step is to hook this up to task scheduler and have it run the first Sunday or something.
Fast forward 10 years and I am still in need of the same thing, as we don't have Vista or Windows 7 deployed. But now I have a much better understanding of what needs to happen and even better I know where to start!
PSExec, which is part of the System Internals PsTools suite is my answer these days. A simple script, run from my PC (still manually for the time being) is able to handle defragmenting all of our PCs.
psexec @C:\Updates\Comps\AppPCs.txt -n 10 -c -f -d JkDefragCMD.exe
I use the files with PC names to speed up re-deployment of everything. The -n 10 flag tells psexec t0 wait 10 seconds before it times out the PC, instead of 60 (I believe this is the default). -c copies the file (jkdefragcmd.exe) to the remote system. The flag -f forces copying, even if the file exists. The flag -d doesn't wait for the process to terminate, this is as asynchronously as I can do.
My next step is to hook this up to task scheduler and have it run the first Sunday or something.
Monday, September 13, 2010
How to Reset the Password(s) on a Linksys SRW248G4 Switch
- Connect to the Serial port on the back of the switch with a serial cable. (Putty on Windows works well)
- The defaults for connecting to the serial port are 38400 baud, 8 data bits, no parity, 1 stop bit, no flow control.
- Confirm that you have an active serial connection by pressing enter a couple of times. You should receive a login screen
- Once you have successfully connected to the serial port restart the switch by unplugging the power. (Either from back or from the power strip, my preferred method is the back)
- The switch will start its POST process.
- Look for the line “Autoboot in 2 seconds - press RETURN or Esc. to abort and enter prom.” (Please note: Do not hold down Esc. or Enter. Only press it once.)
- You will know that you interrupted the boot sequence when you are prompted with a startup menu. Select option 3 “Password Recovery Procedure”.
- The screen will display “The current password will be ignored!”. Press Enter to reboot the switch.
- Once the switch has restarted login with the default admin user and no password. This will work on either the terminal or the web interface. Proceed to create / modify users and passwords. This does not reset the rest of the configuration, just user accounts.
- Once you have your new credentials reboot the switch again. Document the credentials.
- Verify that your new credentials work by logging into either the web interface for the command line interface.
- Disconnect the cable and store it safely.
Monday, April 19, 2010
NDO2DB daemon startup script
I would like to thank Chris over at http://sysengineers.wordpress.com for the excellent post on how to daemonize NDO2DB. His post NDO2DB startup script for RH (EL) / OEL does an excellent job getting everything working. I had to make one small change to the script as I kept my ndo2db executable as ndo2db-3x instead of just ndo2db.
For me this fixes a problem where I have to remember to manually start the ndo2db service after a server restart, which hasn't been happening the past few times. The second way I am going to fix this problem is by creating a checklist of things that need to happen during a server restart / boot up sequence for each PC. This will also reduce the effort needed for when I move the rack later this year. The third way I will be checking this is by adding a check to Nagios to make sure this is running at all times.
This also illustrates my general way of checking / double checking things. One go off of a checklist. The double check is to verify that Nagios is seeing the same thing.
Labels:
CentOS,
Linux,
nagios,
Networking Services,
OS,
programming,
Redhat,
skills,
software,
Tools
Monday, March 15, 2010
3Com Switch find Uptime
We had another incident with our VoIP phones restarting on Friday. As part of my troubleshooting efforts I needed to be 100% certain that the POE switch providing power to our phone didn't reboot.
To start there is no display uptime command in the CLI. But if you run the display version command part of the output is "Switch 4500 PWR 50-Port uptime is 6 weeks, 5 days, 2 hours, 25 minutes". This is exactly what I was looking for!
It should also be noted that the web interface for the switch shows the uptime on the device summary page.
Wednesday, February 10, 2010
Upgrading the Firmware on a 3com 4500 switch
Again, this is mostly for my own notes, but someone else may find it useful. Last year we purchased a 3c0m POE switch for our new VoIP phone system. This year I needed to update the firmware on it, but with only 8MB of flash drive I ran into a few problems.
First off, BACK UP EVERYTHING!
I used the TFTP method to transfer files to and from the switch. I used Solarwinds TFTP Server on my PC.
File name Prefix / Suffix
s3n / .app = 4500 application software.
s30 / .btm = 4500 boot ROM software
s3p / .web = 4500 web file (HTTP management interface)
3comOScfg.def / .def = 4500 config file
So first things first backing up via TFTP:
<4500>tftp [IP OF TFTP SERVER] put flash:/s3004_01.btm
<4500>tftp [IP OF TFTP SERVER] put flash:/s3p04_03.web
<4500>tftp [IP OF TFTP SERVER] put flash:/s3n03_03_02s56p05.app
<4500>tftp [IP OF TFTP SERVER] put flash:/3comOScfg.def
Please change these files as you see fit. Do a dir on the root directory to get the listing for your particular switch.
Now that we have that backed up we need to clean up the flash:/ drive to make room for the updates.
<4500>delete s3004_01.btm
<4500>delete s3p04_03.web
<4500>detete s3n03_03_02s56p05.app
Now we also have to empty the recycle-bin. This is where I got stuck as I didn't know a CLI could have a recycle-bin, or have never seen it done before.
<4500>reset recycle-bin
See as we now have the free space we need to get the new files we can pull them down from the TFTP server.
<4500>tftp [IP OF TFTP SERVER] get s3p02_01.web
<4500>tftp [IP OF TFTP SERVER] get s3o01_01.btm
<4500>tftp [IP OF TFTP SERVER] get s3n03_02_00s56.app
Again, you will need to use the files that are the current firmware update.
One of the last steps is to tell the switch what files to use on next boot.
<4500>boot boot-loader flash:/s3n03_02_00s56.app
<4500>boot bootrom flash:/s3o02_01.btm
Finally we will save the configuration and reboot the switch.
<4500>save
<4500>reboot
Thats it! After the switch reboots you will be running the newest software, except for the .web file. For some reason this is left out of all of the documentation that comes with the update. I didn't write down the commands I used to get it to update, and I honestly don't think they worked.
Labels:
3com,
certification,
Firmware,
OS,
programming,
skills,
software,
Study Tips,
Switches,
Tools
Thursday, December 24, 2009
Locking Down Mozilla Firefox
One of the main draw backs to Mozilla Firefox from most corporate IT people is its inability to be locked down, or taken care of by Group Policy. While you cannot lock down Firefox from within Group Policy alone, you can script these fixes into place to lock settings down.
To start you will need a program to byte shift a config file so that Firefox can read it. I used Byte Shifter.exe. There are also websites that do it all in a browser, but I haven't used them.
You will also need to edit the all.js file in "C:\Program Files\Mozilla Firefox\greprefs\" to include:
pref("general.config.filename", "mozilla.cfg");
I put that line at the absolute bottom, but it might not matter where it goes.
To start create an empty file. I called mine mozilla.txt since the result file will be mozilla.cfg.
The file must start with // on its own line.
Add in any settings you want to lock down. You can peruse about:config for settings, and again after you lock them down to check for the "locked" option.
Below is my mozilla.txt file, with host names changed to protect the innocent. I have included comments (They start with //) to explain things a bit further.
//
//Lock the option for startup page. 0 = "Show a blank page", 1 = "Show my home page", and 3 = " Show my windows and tabs from last time"
lockPref("browser.startup.page", 1);
//Set the home page. Use a pipe to include many home pages as tabs.
lockPref("browser.startup.homepage", "http://www.mysuperawesomecompany.com|https://mail.google.com");
//Set the browser history to something a bit longer than the default 7 days.
lockPref("browser.history_expire_days", 90);
lockPref("browser.history_expire_days.mirror", 90);
// Clean up certain things every time Firefox shuts down. This keeps things clean and running smooth for us, your results may very.
lockPref("privacy.sanitize.sanitizeOnShutdown", true);
//We do not want to clear the history on shutdown.
lockPref("privacy.clearOnShutdown.history", false);
lockPref("privacy.item.history", false);
//We can clean up the downloads history. I have seen things get really slow if this doesn't happen.
lockPref("privacy.item.downloads", true);
//Clear the cache.
lockPref("privacy.item.cache", true);
//Clean up cookies.
lockPref("privacy.item.cookies", true);
//Remove any session info.
lockPref("privacy.item.sessions", true);
//We do not want to keep passwords saved.
lockPref("privacy.item.passwords", true);
//Do not prompt to do this, just do it.
lockPref("privacy.sanitize.promptOnSanitize", false);
lockPref("signon.rememberSignons", true);
//Do not allow the "Show passwords" button.
lockPref("pref.privacy.disable_button.view_passwords", true);
//Don't use a proxy.
lockPref("network.proxy.type", 0);
//We keep one version of Firefox for a while. The newest version breaks things in our application, so we currently need to just run what we have.
lockPref("app.update.enabled", false);
//Disable extensions.
lockPref("config.lockdown.disable_extensions", true);
//Disable themes.
lockPref("config.lockdown.disable_themes", true);
//Show the downloads window when downloading a file.
lockPref("browser.download.manager.showWhenStarting", false);
//Close the downloads window when all downloads are done.
lockPref("browser.download.manager.closeWhenDone", true);
//Save files to:
lockPref("browser.download.useDownloadDir", true);
lockPref("browser.download.dir", "c:\\%homepath%\\Desktop");
lockPref("browser.download.downloadDir", "c:\\%homepath%\\Desktop");
lock{ref("browser.download.folderList", 2);
//Always ask me where to save files.
lockPref("browser.download.useDownloadDir", false);
//Always check to see if Firefox is the default browser.
lockPref("browser.shell.checkDefaultBrowser", false);
//New pages should open in a new window.
lockPref("browser.link.open_external", 2);
lockPref("browser.link.open_newwindow", 2);
//New pages should open in a new tab.
lockPref("browser.link.open_external", 1);
lockPref("browser.link.open_newwindow", 1);
//Warn me when closing multiple tabs.
lockPref("browser.tabs.warnOnClose", false);
//Warn me when openining multiple tabs might slow down Firefox.
lockPref("browser.tabs.warnOnOpen", false);
//Always show the tab bar.
lockPref("browser.tabs.autoHide", false);
//When I open a link in a new tab, switch to it immediately.
lockPref("browser.tabs.loadInBackground", false);
//Block pop-up windows.
lockPref("dom.disable_open_during_load", false);
//Load images automatically. 1 = check 2 = unchecked.
lockPref("permissions.default.image", 2);
//enable JavaScript.
lockPref("javascript.enabled", true);
//Some of the advanced JavaScript options.
//Disable the Advanced Button.
lockPref("pref.advanced.javascript.disable_button.advanced", true);
//Move or resize existing windows.
lockPref("dom.disable_window_move_resize", true);
//Raise or lower windows.
lockPref("dom.disable_windows_flip", false);
//Disable or replace context menus.
lockPref("dom.event.contextmenu.enabled", false);
//Hide the status bar.
lockPref("dom.disable_window_open_feature.status", false);
//Enable Java.
lockPref("security.enable_java", false);
Of course there are others and many more settings, but that covers a good number of them. Also I have only tested this on our version of Firefox, which is 2.0.0.20.
Monday, October 19, 2009
Nagios: check_http, using the --invert-regex option
Some times you want to check that something is running or working correctly and you work out tests for that. Other times you want to know when something is broken and throwing error messages. This is about the latter, a proper HTTP 200 code is great and all, but what if the page is just showing "Too Many Connections" instead of your home page? My old check_http command for this server used to look like, well, check_http. I didn't check anything about it specifically, just that it was returning a 200 code.
Today however I knew I needed something more in depth. Our database server lost its local network connection, but still was available over the public IP, which is what I test against. Once we re-directed the SQL requests to the public IP address of the server everything started working again, until we ran across "Too Many Connections". The database server kept all of the "local" connections open and thus we ate up the rest.
So, how to test for this scenario? After reading through the man pages of check_http I saw this little gem "--invert-regex Return CRITICAL if found, OK if not". This I knew was exactly what I was looking for! If it sees our error codes it will go Critical! Now to put this gem into practice. Here is where the man pages fall short. There is no explanation on HOW to use this, just that it exists. I tried the obvious to me "check_http -H hostname.com -w 3 -c 5 --invert-regex 'Some string'", but that didn't work. OK, lets try "check_http -H hostname.com -w 3 -c 5 --invert-regex='Some string'" nope that errored out with " option `--invert-regex' doesn't allow an argument".
Third times the charm right?
"check_http -H hostname.com -w 3 -c 5 -r 'Some string' --invert-regex '"
# HTTP OK HTTP/1.1 200 OK - 0.355 second response time |time=0.354966s;3.000000;5.000000;0.000000 size=12975B;;;0
Yes, as it turns out third time is the charm. So that got me thinking some more. How can I ensure that the page is rendering correctly, and if it isn't fail but in a specific way?
"check_http -H hostname.com -w 3 -c 5 -r 'Some string I want in my page' -r 'Some string I don't want to see' --invert-regex '"
You can add more than one -r to the check_http command and it will require all of them to be present for the test to pass, and if one of them fails then it will go critical! Perfect!
If you have any more insight into using the check_http command in Nagios I want to hear about it. We are always running into new failure scenarios that we didn't anticipate and I want to know about them before one of my users tells me about it.
Labels:
apache,
CentOS,
DNS,
Fedora Core,
Linux,
nagios,
OS,
programming
Friday, September 11, 2009
Bash: Finding files between two dates in the current directory
Today my boss asked me for a bash command (or script) to find some files between two dates.
Thanks to Jadu Saikia over at Unstableme his post UNIX BASH scripting: Find Files between two dates, I had a starting point.
This will find all files between the two dates (20071019 & 20071121) in this case.
find . -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'Now, if you want just PGP files you would do:
find *.pgp -type f -exec ls -l --time-style=full-iso {} \; | awk '{print $6,$NF}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20071019 && $1<= 20071121 {print $2}'The second request that my boss was looking for with this is the file size, something that was being left out by awk. So we can fix that by updating the command to:
find *.pgp -type f -exec ls -lh --time-style=full-iso {} \; | awk '{print $6,$NF,$5}' | awk '{gsub(/-/,"",$1);print}' | awk '$1>= 20090624 && $1<= 20090901 {print $2,$3}'We added in a $5 to the first awk command, and the final one had $3 added to it. Also I like human readable file sizes so I added -h to the ls command.
Labels:
CentOS,
Debian,
Fedora Core,
Linux,
OS,
programming,
Redhat,
skills,
Study Tips,
Tools,
Ubuntu
Friday, March 20, 2009
error: can't create transaction lock on /var/lib/rpm/__db.000
I got the error message today while upgrading Webmin. The fix for this is pretty simple:
rm -f /var/lib/rpm/__db.0*
rpm --rebuilddb
The first command clears out any of the files that will lock an RPM from running. The second command rebuilds the RPM database.
Then I was able to install Webmin 1.470 from the RPM that I had downloaded with:
rpm -Uhv /Storage/rpmDownloads/webmin-1.470-1.noarch.rpm
Labels:
CentOS,
Fedora Core,
Linux,
programming,
skills,
Tools
Tuesday, February 17, 2009
Please check your autoconf installation and the $PHP_AUTOCONFenvironment variable is set correctly and then rerun this script.
This is a very simple fix if you are running Redhat Enterprise / CentOS 5.2.
yum install autoconf
I came across this while setting up memcache / memcached. I had tried running phpize from the memcach-2.2.4 directory but was getting the error "Please check your autoconf installation and the $PHP_AUTOCONFenvironment variable is set correctly and then rerun this script."
Once I had autoconf installed I was able to finish the install process
./configure
make
make install
I know there are easier ways to do all of this on CentOS, but we are using a newer version of PHP (5.2.6) than came with CentOS (5.1.6), so using yum to install this would not have worked.
Labels:
apache,
CentOS,
Debian,
fedora core 4,
Linux,
programming,
skills,
Tools,
Ubuntu
Tuesday, February 10, 2009
How to: mod_expires & mod_deflate in Apache running on CentOS 5
While trying to improve the performance of our internal workflow I was tasked with setting up mod_deflate and mod_expires, based off of Yahoo's! Best Practices for Speeding Up Your Website.
The nitty gritty of these two is to make sure they are both listed in your httpd.conf file:
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
Since this server is running the default CentOS 5 apache I already had the support I needed. I then continued on to find the documentation online. Both modules support vhost context so that is the route I went with. I added the lines below to each vhost that I wanted the settings to take place for.
ExpiresActive On
ExpiresByType image/gif A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/png A2592000
ExpiresByType text/css A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType application/pdf A2592000
The syntax for the ExpiresByType directive is:
ExpiresByType mime/type TimeInSeconds.
The A2592000 is Access time + one month. If I wanted the file to be cached for one month from modification date I would have done M2592000 instead.
Other handy numbers in seconds:
86400 = One Day
604800 = One Week
2592000 = One Month
31536000 = One Year (365 Days)
157680000 = Five Years
3153600000 = Ten Years
Why might these large numbers come in handy? Well you want to set far future expiration dates for things that don't change often. The catch is if you need to update one of these items with a far future expiration date. You have to change the file name. Modify the case of the filename will work in the short term but really you want to start adding version strings to all of your static files like CSS and JavaScript.
mod_deflate is much easier to setup. Simply add the line:
AddOutputFilterByType DEFLATE text/html text/plain text/xml application/x-javascript text/css
Again based on the mod_deflate documentation you can add this directive to the vhost directives. Please note that this directive can be placed on just about any context level and as such is very powerful. Also note that you should not try to compress image file because they are already compress and it really is just a waste of CPU and time.
Using these two methods we where able to reduce our website on a first visit from 190.7Kb total to 157.3Kb total.
Before:
9.4K HTML/Text
32.1K Javascript
12.9K Stylesheets
0.4K CSS Images
135.9K Images
190.7K Total
After:
3.1K HTML/Text
15.0K Javascript
2.9K Sytlesheets
0.4K CSS Images
135.9K Images
157.3K Total
I also used the yuicompressor to minify our main Javascript and our CSS file.
After:
3.1K HTML/Text
12.3K Javascript
2.2K Sytlesheets
0.4K CSS Images
135.9K Images
154.0K Total
As you can see using mod_deflate to compress the Javascript and CSS is a more substantial savings in terms of bandwidth than trying to minify.
Granted based on my observations if we really wanted to save on bandwidth and make our pages load faster we would reduce the substantial image footprint.
In case anyone is wondering where I culled all of these numbers from I used Y! Slow for Firebug in Firefox.
Monday, January 05, 2009
How-To: Install, Configure, and Import your first SVN (Subversion) repo on CentOS 5
This might only be useful to me, I don't know but at least I will know its out there and when I have to do this again I will know what to do.
Use wget to pull down the latest source.
tar xzfv subversion.1.2.3.4.tar.gz
cd subversion.1.2.3.4/
./configure
make && make install
svnadmin create --fs-type fsfs /path/to/repo
touch /path/to/repo/.htpasswd
htpasswd /path/to/repo username
svn import -m "First Import" --username=username /source/path/ file:///path/to/repo
Setup the apache config file. Since I use named virtual hosts and typically have multple repos here is a sample.
<VirtualHost *:80>
ServerName svn.yourhost.com
DocumentRoot "/var/www/html"
ServerPath /html/
DirectoryIndex index.php index.htm
<Location /svn/repo1>
DAV svn
SVNPath /path/to/repo/repo1
AuthType Basic
AuthName "repo1 Repository"
AuthzSVNAccessFile /path/to/svn-acl-conf
AuthUserFile /path/to/repo/repo1/.htpasswd
Require valid-user
</Location>
<Location /svn/repo2>
DAV svn
SVNPath /path/to/repo/repo2
AuthType Basic
AuthName "repo2 Repository"
AuthzSVNAccessFile /path/to/svn-acl-conf
AuthUserFile /path/to/repo/repo1/.htpasswd
Require valid-user
</Location>
<Location /svn/repo3>
DAV svn
SVNPath /path/to/repo/repo3
AuthType Basic
AuthName "repo3 Repository"
AuthzSVNAccessFile /path/to/svn-acl-conf
AuthUserFile /path/to/repo/repo3/.htpasswd
Require valid-user
</Location>
</VirtualHost>
The /path/to/svn-acl-conf file contains
[repo1:/]
username = rw
[repo2:/]
username = rw
[repo3:/]
username =rw
This file needs to be updated everytime a new repository is added with the appropriate username and permissions.
I am sure I have missed some steps with this, but it's a decent starting point.
Thursday, November 20, 2008
How to post XML code on Blogger
Yes, my last post made me think for a little while. I wanted to post XML code on Blogger, but it could have been HTML for all Blogger cares. What I had to do was encode the code. Rather than do it the tedious way by hand I choose to use a site such as http://centricle.com/tools/html-entities/ http://htmlentities.net/ to encode it for me.
Once I had the code encode and pasted back into Blogger everything worked out as expected.
Tuesday, October 28, 2008
Using nagios to check PHP script execution time
We want to record the execution time of some of our PHP scripts. I want to use nagios to do this but it is proving rather difficult.
Initially I was hoping I could provide some credentials to nagios for it to use and log into our web site, run a set of scripts, and then report back to us how long each script took. Since our site does not use basic authentication I cannot do it the intial way I wanted.
To work around the issues we are having and hopefully NOT have to write our own nagios plug-in here is what I am proposing.
Create a test directory that uses basic authentication so it is protected from the general public and has a set of scripts that we run with the execution time posted near the bottom. We then use the standard check_http plug-in from nagios to search for that string and parse it as normal.
Hopefully this approach to things won't take too long to develop and prototype. Also I hope to report back with a follow up on what I actually did (with code examples) to get this working.
Monday, October 27, 2008
Adding a calender to your BASH login script
You milage may vary, but on CentOS 5.2 I edited my .bash_profile to contain the line:
cal `date +"%m"` `date +"%Y"`This put a nice looking calendar on my screen when I logged in vi ssh. A small addition but handy non the less.Thursday, October 23, 2008
Powershell: Equal Sign vs. -eq
I was just working on a script in powershell and saw the error "Length" is a ReadOnly property. It was from the code if ($args.count = 0) {exit;}. Having "grown up" writing VB code this would have worked. But in powershell you need to do if ($args.count -eq 0) {exit;} instead.
Subscribe to:
Posts (Atom)