Perl’s “quote word” in Groovy

When it comes to embedding a long list of simple text strings in the code, Perl’s “quote word” is very handy:

my @l = qw(a b c d e f g h);

This is the closest equivalent I could find in Groovy:

def l = "a b c d e f g h".split("\\s+").findAll { it.length() > 0 }

You can also use a multiline string :

def l = """
a
b
c
d
e
f
g
h""".split("\\s+").findAll { it.length() > 0 }

The “findAll” avoid to get empty strings at the beginning of the end of the list in case of empty lines or leading and trailing spaces.

Posted in Uncategorized | Leave a comment

Linux serial TTY hanging

Today, I had the problem that my serial TTY (/dev/ttyACM0, serial over USB) was apparently hanging when receiving data.

However, when more data was sent by the other side, the communication would eventually resume. So at first, I though some kind of receive buffering was involved, but I was wrong. The reason was that the “icanon” option of the TTY was set. In this mode, the bytes are interpreted in some way I don’t (want to) understand, that caused the delays in the data transmission. Disabling it fixed the issue.

# stty -F /dev/ttyACM0 -a
speed 9600 baud; rows 0; columns 0; line = 0;

intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^A; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;

werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;

-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon ixoff -iuclc ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke


# stty -F /dev/ttyACM0 -icanon

# stty -F /dev/ttyACM0 -a

speed 9600 baud; rows 0; columns 0; line = 0;

intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^A; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;

werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;

-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon ixoff -iuclc ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig -icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke

Posted in Linux | Leave a comment

How to identify the end of lines used in a text file

It’s pretty easy to find files that have Windows end of lines (CRLF) with the GNU grep:

grep -lUP '\r$'

And if you need to find files with Unix end of lines:

grep -lUP '[^\r]$'

But you may need more, for instance to find out if files have mixed end of lines. Then you should give “file” a try:

$ file mixed unix windows
mixed:   ASCII text, with CRLF, LF line terminators
unix:    ASCII text
windows: ASCII text, with CRLF line terminators

If you need even more information, for instance the number of CRLF and LF in each file, then you could use the following C program (eol-id). It will tell you this:

$ ./eol-id mixed unix windows
mixed LF=3 CRLF=3 VERDICT:MIXED
unix LF=3 VERDICT:LF
windows CRLF=3 VERDICT:CRLF

Here is the code (eol-id.c):

#include <stdio.h>
#include <stdlib.h>

#define CR 0x0D
#define LF 0x0A

int readfile(char *filename) {
    FILE * fptr = fopen(filename, "r");

    if ( fptr == NULL ) {
        fprintf(stderr, "Failed to open %s\n", filename);
        exit(1);
    }

    int current;
    int previous = 0;
    long cr = 0;
    long lf = 0;
    long crlf = 0;
    long lfcr = 0;
    int result = 0;

    do {
        current = fgetc (fptr);
        switch (current) {
            case CR:
                if ( previous == LF ) {
                    lf--;
                    lfcr++;
                    previous = 0;
                }
                else {
                    cr++;
                    previous = current;
                }
                break;
            case LF:
                if ( previous == CR ) {
                    cr--;
                    crlf++;
                    previous = 0;
                }
                else {
                    lf++;
                    previous = current;
                }
                break;
            default:
                previous = current;
                break;
        }
    } while (current != EOF &amp;amp;&amp;amp; ! result );

    fclose(fptr);

    printf("%s", filename);

    int n = 0;
    char *verdict;
    if ( lf > 0 ) {
        printf(" LF=%ld", lf);
        verdict = "LF";
        n++;
    }
    if ( crlf > 0 ) {
        printf(" CRLF=%ld", crlf);
        verdict = "CRLF";
        n++;
    }
    if ( lfcr > 0 ) {
        printf(" LFCR=%ld", lfcr);
        verdict = "LFCR";
        n++;
    }
    if ( cr > 0 ) {
        printf(" CR=%ld", cr);
        verdict = "CR";
        n++;
    }
    if ( n > 1 ) {
        verdict = "MIXED";
    }
    printf(" VERDICT:%s\n", verdict);

    return result;
}

int main(int argc, char **argv) {
    int i;
    for ( i = 1 ; i < argc ; i++ )
        readfile(argv[i]);
    return 0;
}

Posted in Uncategorized | 1 Comment

How to detect a transparent proxy with nmap

On some networks, the outbound traffic to web servers (ports 80 and 443) might be intercepted on the fly by a transparent proxy.
A simple way to try to detect such a proxy with nmap is to run the following command:


nmap -sT cn.pool.ntp.org -p 80

Starting Nmap 6.00 ( http://nmap.org )
Nmap scan report for cn.pool.ntp.org (202.112.29.82)
Host is up (0.00042s latency).
rDNS record for 202.112.29.82: dns1.synet.edu.cn
PORT   STATE SERVICE
80/tcp open http

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

It tells nmap to initiate a standard TCP connection (-sT) with an NTP server that is far way from me, e.g. China (cn.pool.ntp.org), on port 80 (-p 80).

In the output of nmap, we can see that:

  • the connection was successful
  • the network latency is 42ms

It’s certain in this case that there is a transparent proxy, because the typical latency from Europe to China is almost 10 times higher (370ms vs. 42ms), and typically, NTP server don’t have their port 80 often.

The reason why this is working is that this transparent proxy will blindly intercept the connections to web servers independently of their existence. It’s only when the client sends the HTTP headers that the proxy will try to contact the remote server.

If you want to double check, you can ping the same server from the command line, that will give you the latency using ICMP packets (instead of TCP):


$ ping 202.112.29.82
PING 202.112.29.82 56(84) bytes of data.
64 bytes from dns1.synet.edu.cn (202.112.29.82): icmp_seq=1 ttl=46 time=376 ms
64 bytes from dns1.synet.edu.cn (202.112.29.82): icmp_seq=2 ttl=46 time=378 ms

Compare this latency with the one from nmap.

You may also call the same nmap command from a host that is known not to be behind a proxy, and compare the results. If one nmap tells the port is open and another one the port is closed, and no firewall comes into play, then  it’s one more sign for a proxy.

So in this case, we are able to prove the existence of the proxy, but be aware that if the results were negative, it wouldn’t prove its absence. The proxy might well behave in a way that is not detectable using this method, for instance if it contacts the target server before replying to the TCP connection request from the client.

 

Posted in Linux | Leave a comment

What you should know before choosing JIRA

At my current work place, 2 years ago, we needed an application to track the bugs, features and other todo items for several of our projects. We chose JIRA for various reasons:

  • Large feature set
  • Pretty good end user interface
  • Proven stability and reliability
  • Supported by a solid company (Atlassian)

After 2 years of active use and scaling up both in terms of users and supported projects, JIRA does a pretty good job, however, there is one aspect that is such a pain that we would reconsider our choice if we had to do it again. If you are in the evaluation process and considering JIRA, you should definitely know about this…

During these 2 years of use, we found some shortcomings, like:

Key Resolution Summary Created Votes
JRA-3821 Unresolved Priorities per Project and Resolutions per Issue Type 27/May/2004 1948
JRA-1369 Unresolved Reduce JIRA email chatiness 25/Feb/2003 685
JRA-1991 Unresolved Custom fields for Projects 10/Jul/2003 443
JRA-3406 Unresolved Threaded Comments 16/Mar/2004 404
JRA-5006 Unresolved Allow users to watch a project 21/Oct/2004 232
JRA-5493 Unresolved Ability to add watchers during issue creation 14/Dec/2004 564
JRA-6798 Unresolved Allow admins to translate items configurable in the administration section 26/May/2005 265
JRA-8943 Unresolved WYSIWYG / Rich Text Editor 05/Jan/2006 390
JRA-14543 Unresolved Better support for reply emails from Outlook by mailhandlers 28/Feb/2008 122
JRA-22640 Unresolved Filter by fix version release date 29/Oct/2010 162
JRA-24907 Unresolved labels should be case insensitive 24/Jun/2011 420
JRA-28444 Fixed “Add a comment from a non quoted email body” does not set the Strip Quotes to be TRUE 31/May/2012 16
JRA-29069 Unresolved /rest/api/latest/user/search api doesn’t return all values if username is not specified 24/Jul/2012 47
JRA-29149 Won’t Fix Filter out inactive users in the Users list 30/Jul/2012 264
JRA-34423 Unresolved Add the ability to update issues via REST without sending notifications 21/Aug/2013 108
JRA-35449 Unresolved Translation of Custom Field Description 22/Oct/2013 39

At first, we found it great that Atlassian gives us a chance to be able to give feedback, and voted for the existing tickets. Since the number of votes was pretty high, we expected those issues to be addressed quickly. However, after a while, we realized that this is not going to happen. The only updates were pure PR. From my perspective, their updates can be summarized by: “We don’t plan to fix this in a timely fashion, so just deal with it. But please continue to provide feedback that we will kindly ignore”.

If you read the comments of these tickets, you will realize that we are not the only ones frustrated by this situation. Have a look at JRA-1369, JRA-24907 and JRA-14543.

Digging a bit deeper, a pattern emerges: there is a bunch of very old tickets with lots of votes. I think that the 2 following diagrams are showing well how bad the situation is:

2016-01-05 09_53_55-jira-all-tickets.xlsx - Excel

How to read this: the ticket JRA-3821 is almost 12 years old and has gathered 1900 votes (!) since its creation. The request is clearly legitimate, and you would actually even assume that this has always been covered by JIRA. All 1948 voters probably assumed so. Yet, the latest comment from Atlassian is:

Thank you to everyone who has voted or commented on this suggestion. […] Unfortunately, we are not planning on addressing this in the foreseeable future.

Which I personally interpret as:

Dear 1948 customers,  we have absolutely no interest in you or your needs. Deal with it.”

Another diagram showing a bigger picture:

2016-01-05 09_53_32-jira-all-tickets.xlsx - Excel

How to read this: the tickets that are 11 years old have gathered 11000 votes in total since their creation.

From this diagram, it’s clear that Atlassian has accumulated a huge backlog that they fail to process. So the bottom line for you if you consider using JIRA:

  • You should go through the tickets with the most votes and find out if you can live with them never being fixed (or evaluate potential 3rd party plugins)
  • If you find new shortcomings, you should not count on Atlassian to fix them in a timely fashion, however reasonable or obvious they sound, and despite the significant yearly support fees

 

16. Feb. 2016: compare this with how Jetbrains do it with Youtrack…

17. May 2016: here are some alternatives to JIRA:

  • Assembla
  • Axosoft
  • BugZilla
  • Gemini
  • Jixee
  • YouTrack

 

Posted in Uncategorized | 10 Comments

Converting unsigned to signed integers (using Powershell or Excel)

Let’s assume you got unsigned integers, that actually represent signed ones (using 2’s complement). How to get back the original (negative) values?

The formula is pretty easy:

signed = (raw+2^31) % 2^32 - 2^31

In Excel:

=MOD(A1+2^31;2^32)-2^31

In Powershell (2.0):

Function Convert-UInt-To-SInt($input)
{
return ( $inputAsInt + [Math]::Pow(2, 31) ) % [Math]::Pow(2, 32) - [Math]::Pow(2, 31)
}

In Groovy:

(raw + (1L << 31)) % ( 1L << 32) - ( 1L << 31 )

Posted in Uncategorized | Leave a comment

MediaWiki: Search for page content directly in the database with SQL

To search text in the last revision of MediaWiki pages directly in the database with (My)SQL, use the following request:

select p.page_title FROM page p INNER JOIN revision ON page_latest = rev_id INNER JOIN text ON rev_text_id = old_id where old_text like '%text to search%';

This may be useful for instance to find pages which are using a deprecated plugin.

Posted in Uncategorized | Leave a comment

Zimbra: How to delete read-only events from a calendar

Today I needed to delete all the events from a calendar in Zimbra. It seemed straight-forward:

  • open the calendar
  • switch to the list view
  • set appropriate start date and end dates
  • select all
  • hit “Delete”

Unfortunately, Zimbra refused to proceed, because some of the messages were “Read-only”. I didn’t find any way to find out which messages are “Read-only”, let alone to change that.

However, I found a workaround which might be useful if you are in the same situation:

  • Create a new temporary calendar
  • Move all the events from the main calendar to the temporary one
  • Delete the temporary calendar
Posted in Uncategorized | Leave a comment

Shell, wc : getting progress in real time

On Unix/Linux, “wc” is a very useful tool to count the number of lines in a file or a stream. However, sometimes, the file or stream is so big that it takes minutes or longer to get the final result. In such cases, you might want to get feedback regularly, in real-time, about the progress. Here a simple awk script that will do just that, reporting the number of lines every second, and the finaly the final number of lines:

awk 'BEGIN {T=0} (T!=systime()) { printf "%s %s\n",NR,$0 ; T=systime()} END { print NR}'

(Tested with GNU awk)

Posted in Uncategorized | 2 Comments

Android / CyanogenMod : moving contacts between accounts

Contacts in Android are stored in one or more address books. One of them it the “Local” address book.

I needed to move my local contacts to a remote account, using CyanogenMod 12. I found out it’s not obvious, and Google didn’t help. This looks like the most straight-forward method:

  • Start the Contacts app
  • First, export the contacts:
    • In the context menu, select “Contacts to display”
    • Select the source address book
    • In the context menu, select “Import/export”
    • Export to storage
  • Now delete all contacts in the source address book:
    • In the context menu, select “Delete”
    • In the context menu, select “All”
  • Now import your contact in the target address book:
    • In the context menu, select “Contacts to display”
    • Select the source address book
    • In the context menu, select “Import/export”
    • Import from storage

Voilà!

 

Posted in Android | Leave a comment