Margin is too small to contain this

Method history - time dimension of code repositories

2009-02-16T18:50:00.003+03:00

Recently I've been looking into various aspects of integration between various version control systems and Itellij IDEA - one of the best Java IDEs around. Although they did a great job with their new version in terms of performance and feature set, but I feel that VCS is conceptually on the same level where VisualStudio with MS SCC API was decade ago. Yes, JetBrains have added integration of changes between SVN branches, but frankly speaking, do you really need this is your IDE or external tool will work for you?
I feel that we need more fresh ideas in this space. Something that you typically don't expect, out of version control like git-bisect. For instance, it would be much more interesting if I will be able to view revision history on the level of individual methods in Java code, rather than on the file itself. This way I would be quickly answering questions like "who and why wrote this piece of s..t?". This question very often comes to me even if I'm hacking in my own code, not only someone else :) Git-blame is similar, but it does not understand the programming language (which IDEA does very well), so it is operating on the level of lines, rather than meaningful pieces of code.

Telescopic text – a form of lightweight hypertext

2009-02-16T16:24:00.002+03:00

I don’t like helps and software documentation. I’m only looking there if there is no other choice. And I’m getting really frustrated reading it, because typically the assumption that every technical writer is doing about reader’s knowledge is wrong when it’s applied to me (bad luck?). I’m either know too little (so the text is cryptic) or they are trying to explain me everything from the first grade of elementary school so I’m spending ages scrolling down dozens of pages in order to find what I’m looking for.
Wouldn’t it be nice to have a system, which shows you condensed high-level description, and if you don’t know something, you drilling down there? Sounds like a hyper-text and Wikipedia is doing that for ages? Not exactly, when you are clicking on some term, you are leaving the page that you are reading at the moment. Typically reader wants to stay on the same page, but get more details here, right in the text. Recently I found an example of such approach called telescopic text, where you can click on the piece of text and "expand" it. So the technical writer would be creating a text with maximum details assuming that the reader is an idiot. After that one can markup some pieces of it into "collapsible" chunks. Well, it is probably not that simple, because we need normal text on any level, so some form of morphoanalysis might be involved, but it should be doable at least for English. System should be able to identify correlation between "expand" behavior and act proactively – if someone doesn’t what is the "database", we can automatically expand "first normal form" as well. Ideally the system should be able to learn user's level of knowledge in the specific area and present text in the form most conveniently suitable for her (e.g. minimum amount of expand/collapse needed).
Learning might finally get more fun. :)

Web-based IDE for Ruby on Rails

2008-08-09T13:49:00.008+04:00

I was following a couple of web-based IDEs like CodeIDE or ECCO. All of them seem to be more toys rather then actually useful tools. My understanding is that this is partially because of modest project goals and partially - limitations of technology. Let me expand why I believe it can work.
The advantage of web-based IDE is obvious – anyone, who is doing software development for a living knows how many you need to do before start coding. The idea of having web access to already configured environment where you can contribute anytime you want from any browser looks attractive. But the limitations are obvious as well. Do you think it will work in C/++ world? I really doubt that. Even with modern web-based sharing solutions like WebEx that does not sound realistic. Will it work for huge JavaEE project? Well, easy processing of large text amounts in Google Docs or Adobe Buzzword still not comfortable (yet :). We need a technology that produces output, viewable in browser and does not require megabytes of coding. It looks like Ruby on Rails sounds like an ideal choice (especially taking in mind that good RoR IDE is still missing).
Imaging being able to issue all rails commands with nice and clean UI, have a nice refactoring code editor in flash and even ability to debug RoR application. Of cause UI is the most trickiest part – it will not be possible to mimic standalone IDE, we need to do the same tweak that GMail creators did when they were re-thinking classical email client folder tree. But this certainly doable and it would minimize threshold for contribution in open source projects.
It is also very important to provide seamless integration with version control. In fact, the whole development process can be built around tasks, and task descriptions can act as a descriptions for check-ins (and thus provide meaningfull history of changes).
Also I assumes that this would broaden the audience of developers, so we need to make sure that only quality pieces of code gets to the repository. We need to have good automatic checks built-in the system, that prevents check-in of bad code (similair to FxCop in TFS) and we need to provide a possibility to manually review check-ins before integrating them into main branch (that's why git might have advantage over svn)

Feature elimination in RapidMiner

2008-05-25T13:08:00.019+04:00

Recently I've discovered RapidMiner - data mining application. It has quite good, but huge tutorial, where you can read section 3 and 4 to get the impression of what this tool can do for you. Essentially, as input it gets sample data, where each instance is described as a vector of features (e.g. size, weight, color) and indication to which class this instance belongs to. The processing typically consists of pre-processing (converting of feature values in more appropriate format, eliminating features, that does not bring any additional information, etc.), machine learning (it has quite a few algorithms) and post-processing. In the software, you have something similar to IDE, where you can visually construct process from the existing building blocks. Typical problem with this approach is how to map outputs of previous block to inputs of the succeeding one. They seem to choose the most simplest solution, assuming that all outputs has certain type, and there can be only one input/output of certain type. Obviously they were targeting on data mining specialists, so user should not worry about such details.
I was trying to apply it to my task, which essentially is a subset of the whole process. I have a vector of features and I know for sure the way to classify them - take maximum value among features (which is either -1, 0 or 1) and use it as class. I need to get smallest possible subset of features that still get 100% match with initial classification. It appears that there are 2 ways of doing that in RapidMiner:

Directly alter Model file saved on disk - extremely difficult because model file is just a serialized xml-dump of java class, so even the smallest one is 70K
Use ability to generate new attributes based on existing set as a model

And here I faced the problem, which is typical for all WYSIWYG editors, that helps you to build your program from building blocks. As long as you are doing samples - everything work fine. If you need to do one step off-road, you are in dust. I can create a new attribute, but it will have type regular, but all validators expects to see labels which has type predicted(label). And user is not intended to change types of attributes, because it is dangerous.
I'm apologizing if there is a simple way of solving my task, but I think it is a very good illustration of how programming works. If you are constructing a new development environment, and even if you are targeting on non-developers, you need to make sure that there is "advanced" way, where you have no constraints. No one can predict all the cases how your system will be used. Just hide a candy deep enough so only geeks can find it.

Simplify localization contribution for open source projects

2008-04-06T13:14:00.005+04:00

Recently I've been played a bit with VLC player (mostly fixing bugs in ftp and avi modules). Their idea of localization is that someone has to request corresponding .po-file in mailing list and load it in poEdit. I think this is a wrong idea for open source projects. Here you suppose to welcome and simplify contribution of the people, maintaining reasonable level of quality at the same time. A lot of people is willing to donate some of their time for localization on their native language, if setup is not complicated. Aiming to use those, who is willing to spent 15+ minutes can make a real difference in timeline for getting localized versions.
So, first of all, the solution should be web-based (like Rosetta). Second, it should be as simple in setting up/login as possible (username/password/captcha). Third - show one resource-string to be translated at the time. poEdit and other professional translating tools are showing the whole table of "language pairs" - correspondence between phrases on the source and target language. And this make sense if you are targeting on professional translator, not occasional one. Here the whole concept of UI should be build around this "one phrase" translation. In addition to source phrase you can show google translation (using their unofficial api), and translation of phrases, that has some specific terms for the application (to ensure consitent vocabulary across all translation). User should be able to skip this translation and proceed to the next one, or save one's translation, if it passes certain verification rules (such as presents of %d, %s in translated text, which is corresponding to source one, etc)
Ability to perform translation on the level of individual phrases makes it easy to perform simultaneous translation of one file by several people (does not have to merge changes afterwards)

Syncronization of your contact pictures

2008-03-16T12:02:00.013+03:00

Almost any review of the mobile phone features photo of incoming call, where you can see nice and clean picture of the caller. This looks really good, but I've never seen anyone, who is succesfully using this feature. And the reason is very simple - even with the cameraphone it is extremely difficult to maintain a database of quality pictures for your contacts. Especially if you are changing your phone once a year :)
I think this situation opens a niche for the nice piece of software. Someone forwarded me a linke to Spokeo - agregator over multiple on-line services, which is able to present you what are your friends doing in your favourites social networks and services like Picasa. I was amazed how easily this service retrives various information (including pictures) about my buddies from my google address book and social network friend lists. What if someone will create a service, that will be able to update your mobile phone address book with pictures, retrived from LinkedIn, Facebook or VKontakte?

It should be able to operate via OpenSocial API or custom html parsing, shows you a screen where one can manually match your friends and contacts in your address book (of cause system should attempt to do as much automatic matches as possible) and update your phone contact list.

Basically there are 2 possible ways how this can reach user. It can be mobile application, which user can buy for the fixed price and use it on one's phone as many time as possible. This probably will not going to work on j2me, but sounds like ideal case for iPhone (with its centralized software store). Technologically it can be implemented as Windows Mobile application as well, but it will be less appealing for the user

The other option would be to create on-line service with Silverlight front-end. This front-end should be able to communicate with ActiveSync, iTunes, Nokia suite or whatever sync software is installed. There will be no problems with delivering updates for the users. The only reliable way to monetize this service will be paid SMS, because the target audience is 15-25 years old, and they will unlikely visit site frequently (so ads will not work).

Update: Did not do my homework properly: Outsync performs photo-sync between Facebook and Outlook (so it can get to your WindowsMobile afterwards). Fortunately it is .NET based, so someone might adopt it to your favorite social network as well :)

Setup PHP on Xitami

2007-11-09T12:20:00.003+03:00

Recently I had a nessecity to quickly setup PHP development environment. I've read a couple of HOW-TO articles, and since it was not so easy to install IIS on my laptop, I decided to download and install Xitami 2.5 web-server. PHP 5.20 installer claims that it will configure this server automatically, but apparently it does not (I suspect that it was because I've installed everything to %Program Files%). Ok, there was a manual step-by-step instruction, but it also doesn't work very well. At the end of the day, I found the following:

If you specify php-cgi.exe as a filter for .php-extension, it will not work with aliases, only with real directories under your web-root
If you use php.exe, it will work with your aliases, but it $_GET and $_POST will always be empty

So the solution is to use php-cgi together with symlinks.

Fun with web-services and unqualified elementFormDefault

2007-09-06T16:46:00.003+04:00

Recently I was participating in writing client for document-style binding web-service. This is the first time I saw such wsdl in the field, so I immediately wrote a small test application in VS.NET 2005. Although invoke appears to be working normally, I got null response instead of some actual data. I spent a whole week blaming authors of the web service, but they convince me that the problem was on the client side. It was quite difficult to intercept soap messages, because they were using https, but I found an interesting post how you can get them from SoapHttpClientProtocol. Using this small hack, I was able to observe the resulting envelope, that did contains response, which looks legitimate. But at the end in the corresponding C# object I still got nulls. The wsdl begins with the following declaration:

<wsdl:definitions name="testservice"
     targetNamespace="http://www.testservice.com">
<wsdl:types>
 <xsd:schema elementFormDefault="unqualified"
      targetNamespace="http://www.testservice.com/test">
   <xsd:element name="response">
     <xsd:complextype>
       <xsd:sequence>
         <xsd:element name="ident" type="xsd:string">
         <xsd:element name="number" type="xsd:string">
       </xsd:sequence>
     </xsd:complextype>
....

Visual Studio generate me a corresponding class called response with 2 properties, that has Unqualified System.Xml.Serialization.XmlTypeAttribute. Apparently, Axis on the server side was sending me a message, where both ident and number was in http://www.testservice.com/test namespace, which contradicts with the schema definition above. This was the reason why deserialization from XML does produce nulls. At least this time Microsoft proofs to support standards better than Apache Foundation :)

Surviving after StackOverflowException

2007-08-14T13:32:00.002+04:00

In one of my pet projects, there is a necessity to continue execution after StackOverflowException occurs in separate AppDomain. After quick scanning MSDN, I found out that I need to obtain CLRRuntimeHost, get CLRControl after that CLRPolicyManager and finally apply appropriate policy. Of cause you cannot do this trick, when you are inside managed code, so you have to write you own non-CLR c++ module, that apply corresponding policy and run you managed code. It was not clear, how to do this, but I found a good book with variety of samples: "Customizing the Microsoft® .NET Framework Common Language Runtime" by Steven Pratschner. Unfortunately neither of the sources address my particular case with StackOverflowException. I've tried to use SetActionOnFailure() with all documented EClrFailure values, but as soon as StackOverflowException was thrown, the whole process was killed. During my research I found sources of the Common Language Infrastructure (published by Microsoft under Shared Source license), and discovered that there is another undocumented value of EClrFailure called FAIL_StackOverflow. Based on the comment in file eepolicy.cpp in method EEPolicy::HandleStackOverflow() it appears that:

The only possible actions for FAIL_StackOverflow are eRudeUnloadAppDomain and eRudeExitProcess
eRudeUnloadAppDomain will be executed only if domain to be unloaded is non-default

After that everything works like a charm. Usually AppDomainUnloadedException is thrown to the calling code. Below is a snapshot of very simple CLRHost, that specifies action on failure and run specific method in assembly:


int WINAPI WinMain(HINSTANCE hinst, HINSTANCE hprev, LPSTR szCmd, int nCmd){
  ICLRRuntimeHost *runtime;

  HRESULT hr = CorBindToRuntimeEx(
                  L"v2.0.50727",
                  L"wks",
                  STARTUP_LOADER_OPTIMIZATION_SINGLE_DOMAIN,
                  CLSID_CLRRuntimeHost,
                  IID_ICLRRuntimeHost,
                  (void **)&runtime);

  ICLRControl *pCLRControl = NULL;
  runtime->GetCLRControl(&pCLRControl);

  ICLRPolicyManager* pCLRPolicyManager = NULL;
  hr = pCLRControl->GetCLRManager(IID_ICLRPolicyManager,
                                 (PVOID*)&pCLRPolicyManager);

  hr = pCLRPolicyManager->SetActionOnFailure(FAIL_StackOverflow,
                          eRudeUnloadAppDomain);

  runtime->Start();

  DWORD retVal = 0;
  runtime->ExecuteInDefaultAppDomain(L"Test.exe",
                                     L"Test.Program",
                                     L"Start",
                                     NULL,
                                     &retVal);
  return 0;
}

GUI test automation of web applications

2007-04-06T16:52:00.002+04:00

In course of my past projects, I've multiple times tried to establish effective GUI test automation and for some reasons my attempts always failed. It does not mean that automation was not created at all, but very low effectiveness for significant effort. One of the interesting side effect, that I've seen was that complex GUI test automation usually require someone's attention - you cannot just add it into task scheduler at 2 o'clock in the night, and get result in the morning.
I've tried it with the different people (sometime I even assign good developers into it), but still they encountered enormous amount of technical problems. So I've realized that its not the fault of the people, but rather fault of software. No matter whenever you are using cheap AutomatedQA TestComplete or expensive Mercury WinRunner/QTP, the tools are huge, pretend to do everything and but actually did each task with significant limitations. The worst thing about this is that if you encounter a problem, neither google nor support usually cannot help you.
Recently I was thinking about problem of automated periodic posting of some information on various web-sites. We've tried to automate it using TestComplete, but stability and speed was far from perfect. In fact it refuses to work through TaskScheduler at all, and you will need someone, who will manually run corresponding .bat file from time to time. In order to change this situation, I've start thinking about tools like NUnitASP. It looks lightweight and cool, but does not suite my needs, because of the nature of my task. Usually such type of sites are not really happy when someone is using robot to post data there, so they actively resisting it. I need an ability to click on links and fill fields in the actual browser.
So I was thinking - if I need a real browser to automate this task - why cannot I incorporate one in my C# program? Quick search shows me one good sample on codeproject. I've downloaded it and tweaked according to my needs and it works! I've running this on task scheduler completely unattended, and just getting results from there.
I'm going to post technical details and some point of time later, when I will polish them, but the idea of this post is very simple: if you need GUI test automation - consider writing your own tool for this. Of cause it is not free, and you will have to fight with non-trivial MS implementation of DOM, but this is the only thing you will have to fight - your own automation code will be very small and lightweight (mine is less than 300 lines now :)

FindWindowEx() does not always return child window controls

2007-04-03T18:38:00.001+04:00

There was a task to control separate application via button clicks. We've created a small prototype, that can identify top-level window, and after that found necessary control inside this window using FindWindowEx. To my surprise it was not always working as I expected. Sometimes I was getting null instead of handle to the corresponding window. So I decided to create my own version of that method:


private static IntPtr WaitChild(IntPtr parent, string windowClass,
                                string windowCaption) {
 IntPtr result = IntPtr.Zero;
 do {
     Thread.Sleep(10);
     EnumChildWindows(parent, delegate(IntPtr hWnd, IntPtr lParam) {
                              StringBuilder className = new StringBuilder(200);
                              GetClassName(hWnd, className, 200);
                              StringBuilder caption = new StringBuilder(200);
                              GetWindowText(hWnd, caption, 200);
                              if (className.ToString() == windowClass &&
                                  caption.ToString() == windowCaption)
                                  result = hWnd;
                              return true;
                          }, IntPtr.Zero);
 } while (result == IntPtr.Zero);
 return result;
}

[DllImport("user32.dll")]
private static extern bool EnumChildWindows(IntPtr hwndParent,
                           EnumWindowsProc lpEnumFunc, IntPtr lParam);

private delegate bool EnumWindowsProc(IntPtr hWnd, IntPtr lParam);

[DllImport("user32.dll", SetLastError=true, CharSet=CharSet.Auto)]
private static extern int GetWindowText(IntPtr hWnd,
                       [Out] StringBuilder lpString, int nMaxCount);

[DllImport("user32.dll", CharSet=CharSet.Auto)]
private static extern int GetClassName(IntPtr hWnd,
                       StringBuilder lpClassName, int nMaxCount);

Sometimes it takes one or two loops while the window is finally appears on the screen, but it works pretty stable now

Accessing web services over https using own-made certificate

2007-03-12T15:34:00.001+03:00

In most of our products we are extensively using web-services. Some of them are relaying on SSL to encrypt data streams. It works pretty good from the .NET Framework client if corresponding server certificate is issued by certification authority known by Windows, but if you want some $$ in your budget, you probably will generate the certificate by yourself. When you are trying to access such service from C# program, you will get "The underlying connection was closed: could not establish trust relationship for SSL/TLS secure channel". There is a good post about how to make it work by Jan Tielens, but for .NET Framework 2.0 there is even simpler solution - put the line below somewhere during initialization of your application


System.Net.ServicePointManager.ServerCertificateValidationCallback +=
                               delegate { return true; };

How to print multiple copies of page if PrinterSettings.MaximumCopies==1

2007-02-15T11:59:00.002+03:00

I've encountered problem that with all my printers I cannot print several copies of pages using PrinterSettings.Copies=2. .NET Framework just ignore it, and MSDN state why:

You can use the MaximumCopies property to determine the maximum number of copies the printer supports. If the number of copies is set higher than the maximum copies supported by the printer, it will be ignored, not causing an exception.

The problem is that I've never seen the case when MaximumCopies is different than 1. Since this is just a wrapper around GDI DeviceCapabilities(DC_COPIES), I've tried to access it manually via PInvoke, and still get 1
Ok, so how to print multiple copies of the page on the printer, that does not support it? Quite simple:

PrintDocument doc = new PrintDocument();
int remainingPages = /* Number of copies */;
doc.PrintPage += delegate(object sender, PrintPageEventArgs e) {
  e.Graphics.DrawImage(/* Print your page here*/);
  e.HasMorePages = (--remainingPages > 0);
};
doc.Print();

ArgumentException while Saving TIFF as JPEG

2007-02-07T14:46:00.001+03:00

In one of project I'm involving, we are doing some kind of picture processing. If picture comes from the digital camera as .tiff-file, we are doing rotation (if this specified in exif-header) and saving it as high-quality jpeg to conserve disk space. We've discovered that for some pictures we were getting ArgumentException while doing Image.Save(string, ImageCodecInfo, EncoderParameters) MSDN does not mention anything about that, but since the whole image class is just a wrapper around GDI+, we are basically getting error from the corresponding WinAPI function. Funny enough I was able to save exactly this image as bmp, but of cause it does not help us to minimize space usage.
So I started to trace what else we are doing with the image. I found out that we are also stripping all exif information via Image.RemovePropertyItem() because sometimes thumbnails which are stored there can be quite large. And it appears that if we are preserving certain items in exif (namely TIFFTAG_EXIFIFD and TIFFTAG_PHOTOSHOP), image can be saved without any problems.

Be careful when you are specifying hosts in defaultProxy/bypasslist

2007-01-10T20:40:00.001+03:00

In our company all http traffic comes through squid. It is well known that squid has own opinion what is right and what is wrong in http. One good example is that all requests, originated from squid has “HTTP1.0” in header even if originally it was 1.1 Also squid hates DIME, so if you want to use external web-service that utilize DIME, you have to bypass squid. .NET Framework has quite good facilities in specifying proxy settings for the application. In fact when you are specifying hosts in <bypasslist>, you can use regular expressions. I was really surprised when once it doesn’t work in my application. I spent a couple of hours trying to track down what is wrong with the program, and finally discovered that my DIME requests was not bypassing proxy as intended. It appears that I was using ip address in <bypasslist> and domain name in the url of web service. So when .NET framework is trying to much them, it does not perform dns-lookup. Be aware of this!

Awkward behavior of System.Collections.Generic.List.BinarySearch

2006-10-18T15:32:00.001+04:00

I was trying to build my own morphoanalyzer for Russian language. First step is to search through the list of word radicals – prefixes which will remain constant in all word forms like "wom" for "woman"/"women". In Russian I get more 100K+ radicals, so search efficiency is definitely an issue. Since my initial list is presorted, but can contain duplicate radicals, I decided to choose .NET 2.0 Generic List over SortedList/Dictionary and use BinarySearch. One of the surprising results I've got is that BinarySearch (Int32, Int32, T, Generic IComparer) does not always return first occurrence of item, if there are duplicates. I realize this is caused by nature of binary search algorithm. After some time I even found corresponding note in MSDN, but I cannot come up with any reasons why someone might want behavior like that. Combination of List and BinarySearch is the only container that accepts duplicate items, and it is quite obvious that most of the people will use BinarySearch to find all occurrences of items, not the random one. I do realize that it will hurt performance insignificantly, but it will become right-out-of-the-box usable solution. I created my wrapper around BinarySearch, which produce correct results. The assumption is that number of duplicate items is significantly less than total number of items in the collection:


public static int TrueBinarySearch(string prefix, int startIdx) {
  int idx = List.BinarySearch(startIdx + 1, List.Count - startIdx - 1,
                              prefix, null);

  if (idx >= 0) {
    while (idx > startIdx && List[idx].Key == prefix) {
      idx--;
    }
  }

  return ++idx;
}