I recently stumbled upon this while watching the diggnation podcast (episode 201). Check it out:
Quickly remove special characters from file names
September 1st 2010 – Version (2.3 Beta) is available. It adds the following new features:
- A “Dry Run” mode. In this mode the application will only display and log the changes to the file/directory names without actually renaming the files and directories. This gives the user the ability to get an idea about what will be done before taking the plunge.
- Fixes a bug where if a file or directory name contains only special characters, the renaming will fail and the recursive algorithm would try to keep going.
June 24th 2010 – Version (2.2 Beta). It adds the following new features:
- Special characters could be removed from directory names as well.
- It could be run in recursive mode that will allow renaming of all files and/or directories in all the sub-directories.
- Removing all the dots in the file names but the last one that indicates the file extension.
- The underscore is no longer considered a special character and it is not removed from the file names.
A few years ago I wrote a small application to remove all special characters from the file names of all the files in a directory.
Very often I would get a bunch of files that needed to be posted on a website and most of them would contain all kinds of special characters. I got fed up doing it manually- file by file, so I wrote this small app.
This is a Windows application written in C++ and works with win 98 and up. Of course Linux does not need anything like that, since you can do this with a quick one line shell command.
It only works with ASCII file names (sorry if you use any other language than English). I could have just as easily wrote it for UNICODE, but I had no need for that.
So, I decided to share it with anyone who wants to use it:
New v.2.3b: RenameFiles ver.2.3b
Old v.2.2b: RenameFiles ver.2.2 b
Old v.1.0:RenameFiles ver.1.0.
It is just a simple executable and does not need any installation. Keeping it simple is the key here. It also creates a log file in the same directory that the executable is in. The log file keeps track of the original file names and the new file names, so you can always find out what was done.
Alternative ways to transfer files when you do not have ftp or sftp server available
Very often I find myself having to transfer files to remote machines that do not have ftp or sftp server running.
Here are a few methods that I use in those cases:
- Using scp:
The secure copy protocol and command are part of ssh. I am assuming that you have Linux or Unix boxes that you are dealing with. If any of them are Windows computers you need to install ssh servers and clients respectively.
The command looks like this:scp source_file user@IP:pathFor example, if I wanted to copy the 1.txt file to the /tmp directory on the remote computer with ip address of 24.155.21.105 as root, I would do:
scp 1.txt root@24.155.21.105:/tmp/ - Using ssh:
cat source_file | ssh user@IP -c ‘cd /target_dir ; cat > target_file’For example if I wanted to transfer my local file 1.txt to the same remote host and save it in the /tmp directory under the name of 2.txt, I would do:
cat 1.txt | ssh root@24.155.21.105 -c ‘cd /tmp ; cat > 2.txt’ - Using netcat:
First you need to start a netcat server on your local machine to serve the files:
cat source_file | nc -l port_numberThen on the computer receiving the file you do:
nc IP port_number > target_fileFor example, if I wanted to send the same file 1.txt to the same remote computer and save it as 2.txt over port 3030 (you can chose any port, but make sure it is over 1024), I would do:
On the local computer:
cat 1.txt | nc -l 3030On the remote computer, receive the file by specifying the IP address of the machine sending the file:
nc 122.45.62.10 3030 > 2.txtAgain, note here that the ip address specified is of the computer sending the file (unlike the previous examples with scp and ssh).
Find out the subdomains of a given domain name with dig
First, find out the name server(s) for the domain name in question:
dig wikipedia.comLook under the “AUTHORITY SECTION”:
;; AUTHORITY SECTION:
wikipedia.com. 163475 IN NS ns2.wikimedia.org.
wikipedia.com. 163475 IN NS ns1.wikimedia.org.
wikipedia.com. 163475 IN NS ns0.wikimedia.org.In this case wikipedia.com has 3 name servers: ns0.wikimedia.org, ns1.wikimedia.org and ns2.wikimedia.org. Now we can query one of these three servers for the subdomains of wikipedia.com:
dig @ns1.wikimedia.org wikipedia.com axfrHere is what we get back (the list is rather long, so I have truncated it quite a bit);
; <<>> DiG 9.5.0-P2 <<>> @ns1.wikimedia.org wikipedia.com axfr
; (1 server found)
;; global options: printcmd
wikipedia.com. 3600 IN A 208.80.152.2
wikipedia.com. 86400 IN NS ns0.wikimedia.org.
wikipedia.com. 86400 IN NS ns1.wikimedia.org.
wikipedia.com. 86400 IN NS ns2.wikimedia.org.
wikipedia.com. 3600 IN MX 50 lists.wikimedia.org.
wikipedia.com. 3600 IN MX 10 mchenry.wikimedia.org.
aa.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
aa.mobile.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
aa.wap.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
ab.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
ab.mobile.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
ab.wap.wikipedia.com. 3600 IN CNAME rr.wikimedia.org.
...
...
...Note: Not all dns servers will allow axfr protocol queiries. Those will return “Transfer failed”.
Has Apple become like Mirosoft? Or may be even worse?
I love Apple!
I cannot live without my iPod, much less without my iPhone! The “I am a Mac, I am a PC” commercials on TV are so adorable… Sometimes I rewind my DVR just to see one of those commercials again. In these moments I tell myself “Show that evil empire Microsoft how things are done!”.
But then… I must have been brain dead for a while! You see, what got me is that I have always been for a free market and fair competition. And what Microsoft was doing was not right. No two ways about it. Apple was the company that materialized those feelings and stood up to Microsoft. Naturally, I (like many others) fell for it and blindly screamed “Go Apple!”.
It turns out that I have supported a company that became worse than the one it had set out to defy.
Apple said that if you want to have an iPod, you can only use it with iTunes and if you want to buy music for it… guess what?… you need iTunes for that as well. Same applies to the iPhone with the only difference that you have to be an AT&T customer. Oh yeah… and all these songs you might have bought from Apple, you can’t put on any other mp3 player but on an iPod, unless you remove the DRM protection.
On the computer front things do not look much more different. Mac OS X can legally run only on officially approved Apple hardware and vise versa. If you like the OS, well your only chance is to drop some major cash for the hardware as well. This goes over and beyond what Microsoft has ever attempted to do. After all you can run Windows on anything you feel like.
I am a huge believer in FOSS and Apple has shaped to be the absolute opposite of this philosophy. Even though their OS is built on top of FreeBSD and NetBSD, and they have ported a number of Linux applications to Mac OS X, they have not contrubuted back to the community at all. They even officially announced that iTunes will not be released for Linux. I would like to think that the only reason for this is because they know that Linux users are not going to tollerate DRM content anyway.
KARMA on the Fon and Sniffing Wireless Network Traffic with Ubuntu – Step by Step
KARMA is an application that transforms the right wireless NIC into the ultimate Access Point. Unlike a regular AP, which advertises its SSID to whoever wants to connect to it, the KARMA enabled AP passively listens to any client wireless requests and then responds to it with the SSID that it probed for and thus impersonating virtually any Access Point. In short, it presents itself to each client as whatever the client wants it to be and allows it to establish a connection. So it can be “Linksys” to one computer, “MyHome” to another and completely different to someone else.
In order to run KARMA, you need a wireless card with the appropriate chipset, which supports the MadWifi drivers. As a general rule the Atheros based chipset are compatible with MadWifi, but you can check the complete list with the supported hardware just in case before you buy anything.
MadWifi drivers and KARMA are included in the BT3 Linux distribution and that makes it real easy to turn your laptop into the perfect Access Point. Just boot into BT3 from a CD or a USB thumb drive.
KARMA also simulates different services like FTP and DNS so that you can temporarily trick the wireless clients that they are getting somewhere, just like a “honey pot”. Although this is a lot of fun, a lot more exciting is when you reroute them out to the internet so that they can browse just like they expected and at the same time you can examine their network traffic without them even suspecting anything.
In this case it is best to install KARMA on a router. Just make sure it has the right wireless chipset, then blow out the firmware it came with and install DD-WRT, OpenWRT, Tomato or any of the sort, then put KARMA on it and you are good to go. There is a very neat project started by Darren Kitchen and the folks at HAK5, called Jasager. They installed KARMA on a fon router and created a quick web front end to it. In HAK5’s episode # 412 Darren demonstrates network sniffing and session hijacking of the wireless clients connected to Jasager from Windows.
The fon router is relatively quite small and very appropriate for this purpose. So I bought a fonera router myself, put OpenWRT and KARMA on it and continued from there. Check out Darren Kitchen’s step by step process of how to accomplish this.
Lets look at how the network set up will theoretically work. We need to reroute the network traffic from the fon router to the internet and have a packet sniffer like Wireshark in the middle. For this I need to set up a simple gateway on my Ubuntu laptop that will be between the fon router and the Internet. I also have to set up a DHCP server to assign IP addresses to all the wireless clients as they connect to my AP. In this case I will connect the fon router to my Ethernet jack and I will use my wireless NIC on my laptop to connect it to my home router, which gives me the Internet access. Here is a simple diagram of how this will look:
Step 1. Install the DHCP server and the front end to it for easy configuration:
sudo apt-get install dhcp3-server
sudo apt-get install gadmin-dhcpd
The first command above will install the dhcp server and after it finishes it will try to start it and will give you a message that it failed to do so. This is normal, since you have not configured it yet, so just ignore it for now and execute the second command. We will configure it in Step 3.
Step 2. Install a front end graphics tool to set up the gateway:
sudo apt-get install firestarter
Step 3. Configure the DHCP server.
We need to keep the 2 NICs on different networks. In my case the Ethernet card that will be connected to the fon router will be on the 192.168.0.0 network and the wireles NIC that is connected to my home router and then to the Internet is on the 192.168.1.0 network.
Start the GADMIN-DHCPD: System Tools -> GADMIN-DHCPD.
Under “Scope settings” put in the network interface name (in my case it is eth0). You can find out all the interfaces on your computer by running the ifconfig command. Then put in 192.168.0.0 for a “Network address” and finally 255.255.255.0 for the “Subnet mask”. Now click the “Apply” button. You also need to specify the range of IP addresses that the dhcp server can use. So under “Shared IP-addresses ranges” put- “Range from: 192.168.0.10 to: 192.168.0.110”. Then click the “Add” button:
Step 4. Before we can configure our wired interface, we need to give the fon router a static IP address and DHCP server IP. Connect your fon router to your Ethernet port and power it on. Now open your browser and connect to the webif interface of the router. In my case it is http://192.168.1.1/webif.html (Refer to Daren Kitchen’s tutorial if needed). Go to the “Network” tab and change the connection type to DHCP, leave the “Type” to “Bridged” and put in the IP address of 192.168.0.250 and the Subnet mask of 255.255.255.0. Click “Save Changes” and then “Apply Changes” in the bottom right corner of the page. And finally, in the “Connection Type” drop-down go back to Static IP and make sure that the new settings are retained:
At this point you can power off the fon router for now. The next step is to give static IP settings to the wired NIC in order to put it on the same network as the fon router. The 192.168.0.0 network in this case.
Step 5. Configure the Ethernet NIC that the fon router is connected to:
Go to System -> Preferences -> Network Configuration or you can right-click on the networks incon on your menu bar and select Edit Connections:
Then under the “Wired” tab select your interface and hit “Edit”. Then hit the IPv4 tab, select “Manual” from the “Method” dropdown. Hit the “Add” button and put 192.168.0.1 for the IP address, 255.255.255.0 for the Netmask and leave the Gateway blank. You need to also specify a DNS server. There are a hundred different ways you can find your DNS server but the easiest in Linux would be to look at the /etc/resolv.conf file. So, execute the command more /etc/resolv.conf and use that IP address as your DNS server. In my case, the DNS server for my laptop is my router at IP address 192.168.1.1 (you can also use your ISP’s DNS server). In any case, keep in mind that if you take your laptop somewhere else (for example your local coffee shop) the DNS server will change. After you are done, hit “OK”:
Now the wired interface is configured to the 192.168.0.0 network, the DHCPD settings have the same network and interface, and the fon router is also configured on the same network.
Step 6. Power up the fon router. Now you should be able to connect to it on IP address 192.168.0.250. Open up your browser and this time connect to the “Ysager” page. Enter http://192.168.0.250. The Jasager page should come up:
Step 7. Configure the gateway.
The only thing that is left is to configure the gateway so that we can forward the traffic from the wired NIC to the other interface pointing out to the Internet (the wireless NIC in this case, which is connected to the home router and then out to the Internet). Linux has this built into the kernel and we can set all this up using the command line to do the so called “masquerading”, but it is much easier to use a GUI tool that we installed in Step 2.
Fire up the firestarter: Go to Internet -> Firestarter.
First it will ask you to specify your Internet connected device. In this case it is my laptop’s wireless interface, so I chose “wlan0” from the dropdown. Click “Forward” and in the next screen specify your wired Ethernet interface (in this case eth0), then select the checkbox named “Enable Internet connection sharing”:
Next click “Forward” again, and finally click “Save”. This will start the gateway. You should see something like this (only without the vmnet interfaces, those are there because I have VMWare server installed on this machine):
Now enable the DHCP. Click on the “Preferences” button in the above screen. Go to “Network Settings” and select the checkbox “Enable DHCP for the local network” and hit “Accept”:
At this point you can test if your router can find its way to the Internet through your gateway. Just start a terminal session. Type ssh root@192.168.0.250 and enter your password when prompted. After you log in, just ping any Internet site like- ping yahoo.com, you should be getting responses back:
$ ssh root@192.168.0.250root@192.168.0.250’s password:BusyBox v1.4.2 (2007-09-29 07:21:40 CEST) Built-in shell (ash)
Enter ‘help’ for a list of built-in commands.
_______ ________ __
| |.—–.—–.—–.| | | |.—-.| |_
| – || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
KAMIKAZE (7.09) ———————————–
* 10 oz Vodka Shake well with ice and strain
* 10 oz Triple sec mixture into 10 shot glasses.
* 10 oz lime juice Salute!
—————————————————
root@net1:~# ping yahoo.com
PING yahoo.com (68.180.206.184): 56 data bytes
64 bytes from 68.180.206.184: icmp_seq=0 ttl=48 time=85.0 ms
64 bytes from 68.180.206.184: icmp_seq=1 ttl=48 time=84.3 ms
64 bytes from 68.180.206.184: icmp_seq=2 ttl=48 time=86.3 ms— yahoo.com ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 84.3/85.2/86.3 ms
This means that your network settings are correct.
Step 8. Enable KARMA and start Wireshark.
Go back to the Jasager screen you opened in your browser in Step 6 and hit the “Change” button next to “Karma is currently: Off” to start KARMA:
The last thing is to fire up Wireshark and start the capture on the eth0 interface. Now you will be able to see all the traffic of anyone connected to your fon router.
Dynamic Port Forwarding with SOCKS over SSH
Dynamic Port Forwarding with SOCKS over SSH is probably the easiest and cheapest secure method to connect a client application to a remote host over a preferred port.
This method allows an application on a client computer to make request to a local port, then the requests are forwarded to a remote host, which processes the requests and returns the data back to the client. This is very similar to a simple proxy, with the exception that in a proxy set up the application sends data directly to the remote host, while in this configuration we have an emulated local SOCKS server that handles the requests first and then directs them to the remote computer… oh yeah… and the network traffic is encrypted.
Dynamic Port Forwarding over SSH is in a way a simple alternative to VPN.
Why would anyone need this? Two main reasons immediately come to mind. First, if you are on a public, non secure, non trusted, unencrypted network (for example at the local coffee shop) you can use this method to securely connect to a remote host and have your network traffic encrypted and thus protect your data and privacy. Second, if you need to bypass local network restrictions and monitoring services. A good example here is circumventing surf control restrictions. In most cases, surf control is implemented by monitoring the traffic and examining the tcp/ip packets over port 80. By using Dynamic Port Forwarding over SSH, we channel http traffic through a different port and furthermore all http requests/responses are encrypted via ssh and thus cannot be examined or filtered.
Now let’s see how it is done.
Remote Host Setup.
A remote host is needed that will receive and process the requests from the client. Any computer or device can be the remote host as long as it runs ssh. In this respect any Linux box is ideal for that purpose.
From a practical standpoint, your home router is probably one of the best candidates for being your remote host. You need to run open source software on it, like OpenWRT, DD-WRT, Tomato, etc. and have ssh enabled on it. If you are still running the original firmware that you router came with, you are only using a fraction of the hardware’s capabilities and you are missing out on a lot of great features.
If you plan on using a computer on your home network as the remote ssh host, do not forget to set up the port forwarding on your router accordingly.
Client Setup.
If your client is a Linux box, you hardly have to set up anything. All you have to do is connect to the remote host with the following command:
ssh –D port_number user_name@remote_host (for example: ssh –D 8080 dimitar@72.134.54.23)
And then the only thing that is left is to configure your application to use SOCKS. Look at the Firefox setup below as an example (Step 7).
If your client is a Windows box, you have to install an ssh client. PuTTY is a good ssh client for Windows and it is free:
Step 1. Download PuTTY.
Step 2. Open up PuTTY and put in the host IP address or domain name. Leave the port as the default 22 and give the connection a name under “Saved Sessions” and click “Save”:
Step 3. Click the “SSH” on the left hand side under “Connections” and make sure that the “Preferred SSH protocol version” is set to 2.
Step 4. Click on “Auth” and select “Allow agent forwarding”:
Step 5. Go to “Tunnels” and enter the port in the “Source Port” field, select the “Dynamic” radio button and click the “Add” button. Here optionally you can check “Enable X11 forwarding”, which will come in handy if you want to run graphical applications from the remote host. For this piece to work though, you will need an X11 client running on your Windows client, like Cygwin, Reflections X, Humming Bird, etc., but this is beyond this discussion.
Step 6. Click “Session” on the top left to go back to the first screen and save your setting.
Step 7. Now you are ready to configure your application to use the Dynamic Port Forwarding with SOCKS. For this exercise we will configure Firefox. Open Firebox’s “Options” dialog (called “Preferences” if running on Linux) and go to “Advanced” and then “Network”, hit the “Settings” button under “Connection”. In the window that pops up select “Manual proxy configuration”, enter localhost for “SOCKS host” and the port number (8080 for this exercise). Select the “SOCKS v5” radio button and enter localhost, 127.0.0.1 in the “No Proxy for” box:
Now when you browse the internet with Firefox, all the traffic will be directed to the local SOCKS proxy server on port 8080, then the packest will be encrypted and forwarded over ssh on port 22 to the remote host (you can alwasy change the default ssh port to any other port). It is important to notice, that the remote host will decrypt the packets from the client and then make the requests on port 80 without encryption.
Bike Trip from Orlando to Key West (and back)…
Finally… me and a couple of friends got together to do a bike trip for a few days. Somehow the idea of being able to ride bikes from dawn to dusk is irresistible. We packed up tents and a few supplies and took off. On the way down we stopped at lake Okeechobee and the Everglades.
I have to admit that camping is a lot of fun.. but when you have to put up a tent every evening and fold it up every morning, you get a slightly different perspective. A longer trip will require staying at hotels more often than not. Why am I saying this? Well, because an East Coast – West Coast journey seems to be the next step in a logical progression.
Basics of Networking
I have to admit that the majority of the IT professionals I have met have a little or no idea of basic networking. And sadly some of those are sys admins, site admins, programmers and in the extreme cases even WAN and Network admins and Security “experts”. In the above mentioned group, only the programmers can actually be “excused” . Being a programmer myself, I know that you do not have to be aware of what a Level 4 device is in order to write good code. But it helps… especially in troubleshooting scenarios- for example when your program cannot connect to the database due to network or firewall misconfiguration.
I am a visual person and most of the time a picture speaks a lot more to me than a bunch of words, so here is a simple diagram that explains the basics of a TCP/IP network packet:
A little more detailed version of this diagram I drew up in my notebook when I was taking a networking class in college. I still find myself drawing it on a white board when someone asks me a network question.
The “Level” labels above refer to the different layers of the OSI model. It is helpful to think about a network packet as a present, wrapped in multiple boxes. The outermost box corresponds to level 2 in the OSI model (Data Link layer) and it is the one containing the physical addresses of the source and the destination machines. A typical level 2 device is a network switch.
Inside this big box is the level 3 (Network layer) box that contains the IP address of the computer sending the packet and the one that it is directed to. An example of a level 3 device is a router.
The next box is the one that identifies what port the packet originated from and the number of the port that the destination computer should receive the data on. The port number ultimately determines what application is responsible for processing the data. For example, a packet coming on port 21 means that it is an ftp request and that the ftp daemon on the machine should process it. This box corresponds to level 4 on the OSI model – the Transport layer. A PIX firewall is a typical level 4 network device.
And the innermost box of the packet is the “present” itself – the data. This is what was needed to be delivered from one computer to another on the first place and the rest of the boxes were just the means to accomplish this.
The different network devices and computers open the different boxes and read the information they carry as needed. For example, it is helpful to remember that routers strip the Data Link layer to find out the destination (the IP) address of the packet. Then they reconstruct that layer with a new destination physical address – the MAC address of the next hop on the network and with their own MAC address as the source address. This process repeats until the packet reaches its destination.
I have been asked many times by web programmers how to find out the MAC address of the machine that made the original request . As you can see now- this is impossible. The TCP/IP packet that reaches the web server will only contain the MAC address of the last router that processed the frame.
Of course there is a lot more to network packets and protocols than this. But the simple diagram above is good enough to give you a basic idea and to get someone started in the field of networking.
Acer Aspire 4720
I kept postponing the issue of getting a new laptop for quite some time. I would do some research, open a few sites and message boards to see what new is out there, but I was not able to make the final step. I guess I was waiting either for my old one to die or someone to take me by the hand and lead me to the store. Finally, several days ago I came into terms with the fact that neither of these two things would ever happen and I actively started looking.
I did some serious going around some stores and booting different notebooks from my USB thumb-drive with BT3 on it in order to find what I was looking for. It turned out that Acer Aspire 4720-4538 for a little under $500 was ideal for my needs. It has an Intel Dual Core 1.7GHz, 2 GB of RAM, an Atheros wireless chipset and it is light and relatively small.
I recommend it to anyone who is looking for a linux laptop. It comes with Vista, but I blew it out within 5 min. of laying my hands on it. You have to be careful when buying it if you prefer to have a wireless chipset that is supported by the madwifi drivers. Apparently some of these machines come with the Atheros chipset and some don’t. I learned this the hard way. The display model had it, but the one they gave me in the box did not and I had to return it. Both of them had exaclty the same model number. I have no idea why the manufacturer would do something like that.
I am done with buying expensive laptops like I did before. I have a desktop where I can do gaming and CPU/RAM/IO intensive computing, so I do not need a “desktop replacement” laptop. For a $500 price tag I can afford to get a new notebook a lot more oftern if I need to. Come to think of it, if I get a new cheap laptop 18 months from now, at that time it will be faster than almost any expensive laptop I buy now. So, what is the point of spending a lot for a laptop and keeping it for periods of 3 years and longer?