Windows 8 - From Consumer Perspectives

It was an exciting week. Bangladesh cricket team reached the final of the Asia Cup 2012 one-day international (ODI) cricket championship. We enjoyed an exhilarating performances from the boys against some of the best cricket teams in the world. They beat the both finalists of the last ODI world cup and narrowly lost the final to another former world champion Pakistan just by 2 runs. So much enthusiasm was rarely seen in this country in the recent days.

Let's go back to my Windows 8 experience again. In my previous post Windows 8 - Was it the right way?, I tried to figure out if the Windows 8 was the right answer for Microsoft against iPad or Android based tablets.  Since it's still in alpha stage, a lot of changes are expected to appear in the final version. I am concentrating on what I have in my tablet at this moment. I am going to briefly write on the usability of the operating system from the perspectives of different type of users, more specifically end-users and developers.

End-users
The end-users accomplishes usual daily tasks and entertainments.They expect comfortable user interface with stunning graphics. The frequently used programs must be easily locatable and under their fingertips. The Window 8 UI would be an attractive medium if you are already a smartphone user.

Windows 8 presents a very slick user interfaces. The links to the available and frequently usable programs are nicely wrapped all around your screen as large and colorful blocks.  You don't need to click the small Start button at the corner of the screen to run a program. Microsoft has done some really good jobs to tweak the application resolution adjusted to your monitor.
  
The user interface is in reality designed for touch screen. Since now-a-days, many consumers are very much accustomed to smart phone, they might feel comfortable with this interface. If you don't have a touch screen, it's probably better to switch to the classic mode where you have familiar Windows 7 interface.

• All applications run in full screen mode. So you have no concept of maximizing or minimizing the application. MS is somehow backtracking to simpler and neater user experience removing all distracting elements from your sight so that you can concentrate to your real work.
• Only one application is active at a certain period of time. Starting another application will switch the current application to suspended mode where it will use almost no CPU cycles. When there are so many applications in running and suspended modes that the system memory can no more accommodate, older applications are shut down. That said there are some special mechanisms if you have some long running jobs like checking emails or copying large files to a network folder. 

There are only few applications available in the current preview version. All of your favorite applications are still running in classic mode. How many of them appear as metro-style applications is yet to be known. One annoying thing in Metro is that if you start a non-metro application, it will switch back to the classic view.

It is clear that Microsoft has used iPad as example and extended the features in order to streamline the user experience. MS is proud of the Metro and has done a brave job restructuring typical OS interface into smartphone style user interface and introducing this interface in the desktop PCs. The smartphone enthusiasts will definitely welcome that. But from other end-user perspectives, the following questions need to be answered:

• Is Touch the future of  interaction? Will there be an emergence of other interaction/input mechanism? Microsoft has focused exclusively on touch screen influenced by the widespread success of iPhone and Android-based products. But which interaction/input technology should evolve as the dominating medium is the consumers to decide.
• Is only a single application at once is the best approach? Many users run multiple applications simultaneously and drag information from one application to another.
• Is MS delaying the release of Windows 8 as they did with Windows Vista? The competitor is seizing the market rapidly. Apple released the new iPad (previously known as iPad3) on March 7, 2012 and   New iPad tops 3 millions sales only in 3 days - a staggering figure. MS must find a good slot in this year before the market is already saturated with iPad and android-based tablet.

Developers

Those who develop applications for the Windows operating systems belong to this group. Many of the developers also belong to the end-users category as well since they also use their development computer to accomplish day-to-day tasks. Microsoft has done some exciting staff in Windows 8 and Microsoft Visual Studio 11 (VS11) for application development. The most admirable staff is releasing the beta of VS11 and Windows 8 developer preview so that the developers can start with application development. The success of an operating system depends on the number of quality applications available on that system. The public availability of prerelease versions of  them enabled the programmers to develop and deploy their own applications.


The metro style application development has been integrated into the VS11. The most exciting news is that JavaScript is now considered as a first class citizen for metro application software development. You can build applications similar to familiar web programming using JavaScript/HTML5/CSS3 thanks to the Windows Library for JavaScript (WinJS) built on top of Windows Runtime (WinRT), a native API built into the operating system and accessible from C++, VB, C#, JavaScript. 


A lot of enhancements have been made to C++ compiler to comply with the C++11 standard as much as they could. Concurrency is now an integral part of the language. You can access a lot of user interface elements using WinRT which are tedious to program in C++. Microsoft has introduced automatic garbage collection for WinRT objects through several Microsoft-specific C++ extensions.

These are the positive parts of Windows 8 and VS 11 for the developers. The "single application" feature will negatively affect the productivity of the developers most. In this multitasking world, everyone wants to do multiple tasks at the same time. Developers starts C++ compilation, browse the Internet for reference, check email or edit icons all at the same time.  If they need to switch here and there for doing these tasks, frustration will be certain.

As already mentioned, a lot of changes are expected in the final release of Windows 8 and there will definitely be a gigantic marketing efforts in the coming months. MS will compel the OEMs to support their plan and eventually force the consumers to move to the new OS.

Windows 8 - Was it the right way?

So Microsoft has decided to roll out next version of its flagship Windows operating system called Windows 8. The company has reimagined(?) the desktop experience and the result is that the desktops should be looked and used like tablets or smart phones. Since the introduction of Windows 8 Consumer Preview in the last year's build event, I attended several of the conferences, TechTalks, InfoDays organized by Microsoft and partners. In every event one question was frequently asked if the strategy to push all the desktop users to tablets was right.

Apple and Google are snatching up consumer market share from Microsoft and they needed a quick and solid response. For many years Microsoft led the innovation in the desktop PCs and notebooks domains and I was expecting such innovative responses. But what we see in Window 8 consumer preview is what the market already experienced in iPad, Andriod Tabs and in its own Windows Phone.

Once Microsoft was at the forefront of innovation in the IT industry. They made the desktop PC popular and revolutionized the computer industry. But now I felt a huge lack of innovation in MS's offerings like Windows 8. The Tile-Style user interface that Windows 8 has embraced is already in tablet PCs like iPad and Android Tabs. The same interface is already available in Linux based operating system Chrome OS  that Google introduced on July 08, 2009 to the Internet community and later in November 19, 2009 announced the availability of the open sourced Chromium OS. Even the Ubuntu's Unity desktop, first implemented in the netbook edition of Ubuntu 10.10, had tile-style gadgets and application icons.

As a long-term developer of Microsoft's system, I know how aggressively it will push the new OS to the consumers. The company will go in an all-out attack to force the OEM vendors, channel providers, developers, and above all the consumers to buy its products. We provide Windows based-systems to our clients, and know MS will stop the support for the older OS to push their new OS. So called Vendor lock-in forced us once to ship systems equipped with Windows Vista when the OS was not matured enough. That was a terrible experience for us. The complete overhaul of the Windows OS definitely poses a risk for my company to falling into the same trap.

I received my Samsung Series 7 11.6" Slate only last week. My evaluation for the Windows 8 is still under way. This post is just a hint of the psychological adjustments for the developers and end users who were so much accustomed to the desktop pc for so many years. 

keensocial.com has been discontinued

My keensocial.com has been deactivated. Some of my blog posts will not work properly since they need files from keensocial.com. I'm terribly busy now, so will not be able to fix it in the next days. Sorry for the inconvenience.

Defining a file format using XML and XML Schema (XSD) in C#/Java - V

The complete implementation of XML file format is divided into the following steps:

1. XML Schema (XSD) definition
• Let an extension for my file type: .xef (XML example file-type)
• Generate an XML file with data compatible to previously defined schema
2. Validate my XML with the schema
3. Compress huge data fragments
4. Encrypt sensitive data
5. Implementation in Java (Compliance with XSD 1.1)

In the previous four posts (I, II, III, IV), we developed an XML file format on .Net platform. While developing a new application using this technique I needed some advanced features of XSD 1.1 like Type Alternatives and Assertion which are not yet supported by Microsoft.

The type alternatives provides a mechanism to choose a particular type for an element based on the value of an attribute. The element xs:alternative does the trick. Following is an XSD snippet for alternative data type:

<xs:element name="Animal" type="AnimalType">
    <xs:alternative test="@kind eq 'dog'" type="DogType" />
    <xs:alternative test="@kind eq 'cat'" type="CatType" />
</xs:element>

<xs:complexType name="AnimalType">
    <xs:sequence>
        <xs:element> name="Name" type="xs:string" />
        <xs:element> name="Talk" type="xs:string" />
    </xs:sequence>
    <xs:attribute name="kind" />
</xs:complexType>

<xs:complexType name="DogType">
    <xs:complexContent>
        <xs:extension base="AnimalType">
            <xs:sequence>
                <xs:element> name="Veterinarian" type="xs:string" />
            </xs:sequence>
        </xs:extension>
    </xs:complexContent>
</xs:complexType>

<xs:complexType name="CatType">
    <xs:complexContent>
        <xs:extension base="AnimalType">
            <xs:sequence>
                <xs:element> name="Owner" type="xs:string" />
            </xs:sequence>
        </xs:extension>
    </xs:complexContent>
</xs:complexType>

The corresponding XML can be the following:

<Animal kind="dog">
    <Name>Woof</Name>
    <Talk>Woof! Woof!</Talk>
    <Veterinarian>Mike</Veterinarian>
</Animal>

Assertion, as the name suggests, is a mechanism to put constraints on a particular element. It is achieved through the xs:assert element.

<xs:complexType name="DogType">
    <xs:complexContent>
        <xs:extension base="AnimalType">
            <xs:sequence>
                <xs:element> name="Veterinarian" type="xs:string" />
            </xs:sequence>
        </xs:extension>
    </xs:complexContent>
    <xs:assert test="(Talk eq 'Woof!') or (Talk eq 'Woof! Woof!')" />
</xs:complexType>

While validating my XML file with the new schema using my C# program, the validation was failing. After some investigation I have found that Microsoft doesn't support XSD 1.1 yet. So I have to look for alternatives. Apache Xerces-C++ didn't implement XSD1.1 support yet, but Xerces-J did.

After some googling in the Internet, it was clear that Java 7 has taken over the implementation from Xerces-J. I created a Java desktop application in Eclipse and recreate the C# program. Compressing and encrypting were also done using (GZIPOutputStream) java.util.zip and Apache security (com.sun.org.apache.xml.internal.security) packages respectively. To validate against XSD1.1 you need to use SchemaFactory and all your source schema to this factory. Following code snippet shows how to validate and parse an XML file:

DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = builderFactory.newDocumentBuilder();
Path xmlFile = Paths.get(fileName);
Document xmlDoc = null;
try (BufferedInputStream in = new BufferedInputStream(Files.newInputStream(xmlFile))){

 SchemaFactory sf = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
 sf.setErrorHandler(new ErrorHandler() {

  @Override
  public void error(SAXParseException arg0) throws SAXException {
   System.out.println("Error occurred: " + arg0.getMessage());
  }

  @Override
  public void fatalError(SAXParseException arg0)
    throws SAXException {
   System.out.println("Fatal error occurred: " + arg0.getMessage());
  }

  @Override
  public void warning(SAXParseException arg0) throws SAXException {
   System.out.println("Warning: " + arg0.getMessage());
  }
 });

 //StreamSource schemaDocument = new StreamSource(schemaName);
 //Schema s = sf.newSchema(schemaDocument);
 
 // we want to validate against the following schemas; already existing in our res folder
 sf.setResourceResolver(new ResourceResolver());
 Schema s = sf.newSchema(new StreamSource[] { 
   new StreamSource(this.getClass().getResourceAsStream("res/xmldsig-core-schema.xsd"), "xmldsig-core-schema.xsd"),
            new StreamSource(this.getClass().getResourceAsStream("res/xenc-schema.xsd"), "xenc-schema.xsd"),
            new StreamSource(this.getClass().getResourceAsStream("res/set.xsd"), "set.xsd")
   }
 );     
 
 Validator v = s.newValidator();
 StreamSource instanceDocument = new StreamSource(fileName);
 v.validate(instanceDocument);
 
 xmlDoc = builder.parse(in);
  
} catch(SAXException | IOException e) {
  e.printStackTrace();
  System.exit(1);
}

The set.xsd file has been extended to work with compressed and encrypted elements.

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:xenc="http://www.w3.org/2001/04/xmlenc#" elementFormDefault="qualified">

  <xsd:import namespace="http://www.w3.org/2001/04/xmlenc#"></xsd:import>

  <xsd:element name="XmlFileFormat">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="Value1" type="xsd:integer"></xsd:element>
        <xsd:element name="Value2" type="xsd:string"></xsd:element>
        <xsd:element name="Value3" type="xsd:float"></xsd:element>
        <xsd:element name="Settings2">
          <xsd:complexType>
            <xsd:sequence minOccurs="0" maxOccurs="1">
              <xsd:element name="ValueX" type="xsd:integer">
              </xsd:element>
              <xsd:element name="ValueBulks">
                <xsd:complexType mixed="true">
                  <xsd:sequence>
                    <xsd:element name="ValueBulk" type="xsd:float"
                        minOccurs="0" maxOccurs="unbounded"/>
                    <xsd:element ref="xenc:EncryptedData" minOccurs="0" maxOccurs="1" />
                  </xsd:sequence>
                  <xsd:attribute name="Compress" use="optional" type="xsd:boolean" />
                  <xsd:attribute name="Encrypt" use="optional" type="xsd:boolean" />
                </xsd:complexType>
              </xsd:element>
            </xsd:sequence>
          </xsd:complexType>
        </xsd:element>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>
</xsd:schema>

The complete source code can be downloaded from the following link:
http://keensocial.freeiz.com/blogs/xmlfileformat/xmlfileformat-j.zip

Advanced Developer Conference

Last week, I attended the "Advanced Developers Conference 2001" in Frankenthal, Germany. There were some interesting presentations on upcoming version of .Net Framework 4.5, Visual Studio 11 and Windows 8. Since only the preview versions of these products are currently available, the speakers were forced to only present overviews and expected features.

Among nearly 40 presentations from renowned .Net speakers in Germany, I have listened to the following:

KEYNOTE: Cloud-Strategie und Zukunft der Cloud (Cloud Strategy and the future of cloud)
Christian Liensberger



Trennung tut weh: UI und Anwendung mit MVVM und Flow Design trennen (Separation hurts: Separate UI and Application with MVVM and Flow Design)
Stefan Lieser



Parallelisierung - Einführung (Parallelization - Introduction)
Parallelisierung - am Beispiel (Parallelization - Example)
Bernd Marquardt

Parallelisierung - im User Interface (Parallelization - in User Interface)
Bernd Marquardt

Garbage Collector in .NET 4.5 - Teil 1 (Garbage Collector in .NET 4.5 - Part 1)
Garbage Collector in .NET 4.5 - Teil 2 (Garbage Collector in .NET 4.5 - Part 1)
Patric Boscolo

Visual Studio 11 Application Lifecycle Management
Christian Binder, Neno Loje

Neues von der Microsoft BUILD-Konferenz: Ausblick auf Windows 8 (New from the Microsoft BUILD-Conference: Prospects for Windows 8)
Hannes Preishuber

Was ist neu in Visual Studio 2011 und .NET 4.5 - Preview (What is new in Visual Studio 2011 and .Net 4.5 - Preview)
Dr. Holger Schwichtenberg, Neno Loje      


Software Design mit C# 4 - Patterns und fortgeschrittene Sprachelemente (Software Design with C#4 - Patterns und forg
Manfred Steyer

.NET Deep Dive
David Tielke  

Entwickeln von "Metro-Style-Apps" für Windows 8 (Development of "Metro-Style-Apps" for Windows 8)
Hannes Preishuber 

Testbare Systeme mit Dependency Injection und Mocking Framework (Testable System with Dependency Injection and Mocking Framework)
Manfred Steyer  

10 Minute Guide to DirectX 11 Programming in C++

DirectX is already 16 years old. There are tons of resources available online for DirectX programming. This post will give you just some idea how quickly a simple Direct3D based application can be built. This will serve as a starting point for my future posts on XNA Game Studio or SlimDX library for managed 3D visualization on .Net platform.

Prerequisites
1. DirectX Software Development Kit. It will include DirectX End-User Runtimes (June 2010) which is necessary to run your DirectX application on client computer.
2. Some knowledge of COM programming
3. Visual C++ 2010 Express IDE. If you are comfortable with building C++ application from command line, you don't need this IDE.

Concepts

1. Direct3D: Used for all 3D graphics in DirectX
2. Rendering pipeline: DirectX processes build the scene in multiple layers called rendering pipeline.

3. Swap Chain - This is ideally buffers for the rendering. Double buffering with 1 front buffer and 1 back buffer is a popular technique to reduce flicker effect.
   
4. Rendering: Send your buffer to the output.
   

Implementation

1. Create a Win32 Project using Visual C++ 2010 Express "New Project" Wizard like the following and let's name the project RenderDirectX:
   
   
   
   
   
   
   
2. Right click your project in the solution explorer and select properties. Select to C/C++ from the tree and add <directx-installation-dir>\Include to "Additional Include Directories" Field. Replace <directx-installation-dir< with the installation directory of your DirectX.
   
   
   
   
   
3. Now select Linker from the tree in the property pages and add <directx-installation-dir>\Lib\X86 to "Additional Library Directories" Field. Replace <directx-installation-dir> with the installation directory of your DirectX.
   
   
   
   
   
4.  Now select Linker->Input and add d3d11.lib;d3dx11.lib;dxerr.lib; to "Additional Dependencies" Field.
   
   
   
   
   
5. Include the following headers in RenderDirectX.cpp file:
   

   #include<d3d11.h>
   #include<d3dx11.h>
   #include<dxerr.h>
   

6. Initialize the Direct3D device and swap chain with intended feature level and driver types:
   

   ID3D11Device* piD3dDevice;
   D3D_FEATURE_LEVEL eFeatureLevel;
   ID3D11DeviceContext* piD3dContext;
   D3D_FEATURE_LEVEL aFeatureLevel[] =
   {
           D3D_FEATURE_LEVEL_11_0,
           D3D_FEATURE_LEVEL_10_1,
           D3D_FEATURE_LEVEL_10_0
   };
   
   DXGI_SWAP_CHAIN_DESC stSwapChainDesc; // initialize the swap description 
   IDXGISwapChain* piSwapChain;
   D3D11CreateDeviceAndSwapChain( 0, D3D_DRIVER_TYPE_HARDWARE, 0, 0, aFeatureLevel, 
            ARRAYSIZE(aFeatureLevel),  D3D11_SDK_VERSION, &stSwapChainDesc, 
            &piSwapChain, &piD3dDevice, &eFeatureLevel, &piD3dContext );
   

7. Create a render target view and set it to the device context:
   

   ID3D11Device* piD3dDevice;
   ID3D11Texture2D* piTexture;
   piSwapChain->GetBuffer(0, __uuidof( ID3D11Texture2D ),    ( LPVOID* )&piTexture);
   ID3D11RenderTargetView* piTargetView;
   piD3dDevice->CreateRenderTargetView(bufferTexture, 0, &piTargetView);
   piD3dContext->OMSetRenderTargets( 1, &piTargetView, 0 );
   //Bind viewports to rasterizer stage of pipeline
   piD3dContext->RSSetViewports(1, &viewport);
   

8. Change your windows message so that it can update the screen even if no event occurs and draw your image:
   

   ID3D11Device* piD3dDevice;
   while( msg.message != WM_QUIT )
   {
           if( PeekMessage( &msg, 0, 0, 0, PM_REMOVE ) )
           {
               TranslateMessage( &msg );
               DispatchMessage( &msg );
           }
           else
           {
               // Update and Draw 
               float afColor[4] = { 1.0f, 0.0f, 0.0f, 1.0f };
               piD3dContext->ClearRenderTargetView(piTargetView, afColor);
               piSwapChain->Present(0, 0);
           }
   }
   

9. Uninitialize the device and release resources
   

   ID3D11Device* piD3dDevice;
   piTargetView->Release( );
   piSwapChain->Release( );
   piD3dContext->Release( );
   piD3dDevice->Release( );
   

This is rather quick and dirty way of coding with no error handling and object orientation. For real application you should proceed in more object oriented way and use smart pointers to automatically release the resources. DirectX programming is complex and one needs years of experience to master it due to multidisciplinary knowledge required to create attractive visualization effects. To render even a simple content, you have to understand Geometry, Vertex & Vertex Buffer and Pixel Shader. In future posts, I will give more insights into the wonderful world of Visualization, Animation, Gaming on DirectX.

I hope this tutorial will give you a starting point to play with 3D visualization on Windows platform. The complete code for this post can be downloaded from http://keensocial.freeiz.com/blogs/directx/renderdirectx/renderdirectx.zip

Defining a file format using XML and XML Schema (XSD) in C#/Java - IV

The complete implementation of XML file format is divided into the following steps:

1. XML Schema (XSD) definition
• Let an extension for my file type: .xef (XML example file-type)
• Generate an XML file with data compatible to previously defined schema
2. Validate my XML with the schema
3. Compress huge data fragments
4. Encrypt sensitive data
5. Implementation in Java (Compliance with XSD 1.1)

In the previous posts I, II, and III we defined an XML schema and generated XML files which complied with the schema, read/write the XML file with C# programming language and finally, compressed bulk dataset into ASCII-encoded string of binary data. Quite often we have sensible data which is not intended to be published for viewing or modification. In today's post, we will encrypt this data with our private key, so that only the application can decrypt and understand the real meaning of the data. Data encryption is also an integral part of secure exchange of structured data.

Encryption
W3C has a recommendation for encryption of data and representing the data in XML which is hosted at XML Encryption Syntax and Processing. The recommendation is independent of encryption algorithm which is left to the implementing application.

.Net framework provides encryption facilities in System.Security.dll assembly which includes several classes in System.Security.Cryptography namespace for general purpose encryption and some classes in System.Security.Cryptography.Xml namespace to save this encrypted data to XML complying with W3C recommendation for XML encryption.
In our encryption, we will use symmetric encryption algorithm. The .Net implementation of this algorithm resides in RijndaelManaged class. It needs a secret key and an initialization vector for the symmetric algorithm. You can generate a secret key in this web address: https://www.bigbiz.com/genkey.html

For a detailed description of the XML encryption and decryption facilities look at the following MSDN documentation:
http://msdn.microsoft.com/en-us/library/sb7w85t6.aspx

private string m_encrpytionKey = "Wx8Ti1pc0pX"; 
public void Encrypt(XmlElement element) 
{ 
 byte[] encryptionKey = Encoding.ASCII.GetBytes(m_encrpytionKey);
 byte[] iv = Encoding.ASCII.GetBytes(m_encrpytionKey);
 Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(m_encrpytionKey, iv);
 iv = keyGenerator.GetBytes(16);
 encryptionKey = keyGenerator.GetBytes(32);
 RijndaelManaged key = new RijndaelManaged() { Key = encryptionKey, IV = iv };
 EncryptedXml eXml = new EncryptedXml();

 byte[] encryptedElement = eXml.EncryptData(element, key, false);
 EncryptedData edElement = new EncryptedData();
 edElement.Type = EncryptedXml.XmlEncElementUrl;

 string encryptionMethod = null;
 if (key is Rijndael)
 {
  switch (key.KeySize)
  {
  case 128:
   encryptionMethod = EncryptedXml.XmlEncAES128Url;
   break;
  case 192:
   encryptionMethod = EncryptedXml.XmlEncAES192Url;
   break;
  case 256:
   encryptionMethod = EncryptedXml.XmlEncAES256Url;
   break;
  }
 }
 else
 {
  // Throw an exception if the transform is not in the previous categories
  throw new CryptographicException("The specified algorithm is not supported for XML Encryption.");
 }

 edElement.EncryptionMethod = new EncryptionMethod(encryptionMethod);
 edElement.CipherData.CipherValue = encryptedElement;
 EncryptedXml.ReplaceElement(element, edElement, false);
}

Decryption uses the same key. The technique is to load the XML fragment with EncryptedXml.DecryptData method and then replace the data of encrypted XML element with this decrypted data.

public void Decrypt(XmlElement encryptedElement, SymmetricAlgorithm Alg)
{ 
 // Check the arguments.
 if (encryptedElement == null)
  throw new ArgumentNullException("encryptedElement");

 byte[] encryptionKey = Encoding.ASCII.GetBytes(m_encrpytionKey);
 byte[] iv = Encoding.ASCII.GetBytes(m_encrpytionKey);
 
 Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(m_encrpytionKey, iv);
 iv = keyGenerator.GetBytes(16);
 RijndaelManaged key = new RijndaelManaged() { Key = encryptionKey, IV = iv };
 
 // If the EncryptedData element was not found, throw an exception.
 if (encryptedElement == null)
 {
  throw new XmlException("The EncryptedData element was not found.");
 }
 
 // Create an EncryptedData object and populate it.
 EncryptedData edElement = new EncryptedData();
 edElement.LoadXml(encryptedElement);
 
 // Create a new EncryptedXml object.
 EncryptedXml exml = new EncryptedXml();
 
 // Decrypt the element using the symmetric key.
 byte[] rgbOutput = exml.DecryptData(edElement, key);
 
 // Replace the encryptedData element with the plaintext XML element.
 exml.ReplaceData(encryptedElement, rgbOutput);
}

We will encrypt the following XML fragment:

<Settings2>
    <ValueX>30</ValueX>
    <ValueBulks Compress="false">
        <ValueBulk>.0</ValueBulk>
        <ValueBulk>1.0</ValueBulk>
        <ValueBulk>2.0</ValueBulk>
        <ValueBulk>3.0</ValueBulk>
        <ValueBulk>4.0</ValueBulk>
        <ValueBulk>5.0</ValueBulk>
        <ValueBulk>6.0</ValueBulk>
        <ValueBulk>7.0</ValueBulk>
        <ValueBulk>8.0</ValueBulk>
        <ValueBulk>9.0</ValueBulk>
    </ValueBulks>
</Settings2>

Our encryption method generates the following XML fragment:

<Settings2>
    <ValueX>30</ValueX>
    <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element"  
            xmlns="http://www.w3.org/2001/04/xmlenc#">
        <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
        <CipherData>
        <CipherValue>zVymBnB8d344SyxPMAonVj ... HlVxsxupUNg=</CipherValue>
        </CipherData>
    </EncryptedData>
</Settings2>

The complete code can be downloaded from: http://keensocial.freeiz.com/blogs/xmlfileformat/xmlfileformat.zip. The project file can only be opened in Visual Studio 2010. If you have other versions of Visual Studio, you have to create a new solution and add the extracted files from the zip file to the solution.

Defining a file format using XML and XML Schema (XSD) in C#/Java - III

The complete implementation of XML file format is divided into the following steps:

1. XML Schema (XSD) definition
• Let an extension for my file type: .xef (XML example file-type)
• Generate an XML file with data compatible to previously defined schema
2. Validate my XML with the schema
3. Compress huge data fragments
4. Encrypt sensitive data
5. Implementation in Java (Compliance with XSD 1.1)

Compressing Data
In the first post of this series we have defined an XML schema and created an example file complied with the schema while in the second post we have written C# code to manipulate this XML file and validate it with the predefined schema.

 The tag-based verbose representation of XML files was a matter of debates due to increasing overhead during serialization and transfer over networks. There were number of groups involved in addressing and effectively overcoming this problem. W3C Efficient XML Interchange (EXI) Working Group has only recently published the recommendation for binary coded XML on March, 2011. There are only few well developed tools which implemented this recommendation. Two open source projects, both programmed in Java, are actively working on it - EXIficient and OpenEX. efficientXML of Agile Delta is a popular XML compression commercial alternative.

In today's post, we will use some general compression mechanisms like Zip to compress large datasets into binary datasets, encode them to Text form and save into the XML file. The problem with such compression is that you need to rework the existing schema otherwise document might be incompatible with the schema. The following code snippet will return the compressed XML fragment as a string:

public string GetXmlFragmentForValueBulks()
{
    XmlDocument xmlDocument = new XmlDocument();
    XmlElement xmlElementCompress = xmlDocument.CreateElement("ValueBulks");
    xmlDocument.AppendChild(xmlElementCompress);
           
    m_bulk.ForEach(f =>
    {
        XmlElement xmlElementChild = xmlDocument.CreateElement("ValueBulk");
        xmlElementChild.InnerText = f.ToString("#.0", Thread.CurrentThread.CurrentUICulture);
        xmlElementCompress.AppendChild(xmlElementChild);
    });

    if (m_compress)
    {
        byte[] byteArray = Encoding.Unicode.GetBytes(xmlElementCompress.InnerXml);
        MemoryStream stream = new MemoryStream(byteArray);
        MemoryStream outStream = new MemoryStream();
        using (GZipStream compression = new GZipStream(outStream, CompressionMode.Compress, true))
        {
            compression.Write(byteArray, 0, byteArray.Length);
            compression.Close(); // we must close it
            byteArray = new byte[outStream.Length];
            outStream.Position = 0;
            outStream.Read(byteArray, 0, byteArray.Length);
        }
        outStream.Close();
        return Convert.ToBase64String(byteArray);
    }

    return xmlElementCompress.InnerXml;
}

Now load the compressed XML fragment and add it to the document in uncompressed form:

XmlDocument xmlDocument = new XmlDocument();
xmlDocument .Load(fileName);

XmlElement xmlElementChild = xmlDocument.SelectSingleNode("ValueBulks") as XmlElement;

byte[] byteArray = Convert.FromBase64String(xmlElementChild.InnerText);
MemoryStream instream = new MemoryStream();
MemoryStream stream = new MemoryStream(byteArray);
stream.Position = 0;
using (GZipStream decompression = new GZipStream(stream, CompressionMode.Decompress))
{
    decompression.CopyTo(instream);
}
byteArray = new byte[instream.Length];
instream.Position = 0;
instream.Read(byteArray, 0, byteArray.Length);
value = Encoding.Unicode.GetString(byteArray);
xmlElementChild.InnerXml = value;

The following uncompressed XML fragment (... shows for continuity):

<Settings2>
    <ValueX>30</ValueX>
    <ValueBulks Compress="false">
        <ValueBulk>.0</ValueBulk>
        <ValueBulk>1.0</ValueBulk>
        ...
        ...
        <ValueBulk>98.0</ValueBulk>
        <ValueBulk>99.0</ValueBulk>
    </ValueBulks>
</Settings2>

will be converted to the following compressed fragment:

<Settings2>
    <ValueX>30</ValueX>
    <ValueBulks Compress="true">H4sIAAAAAAAEAOy9B2A...ppD8CFQAA</ValueBulks>
</Settings2>

My upcoming post will be last in this series where I will write about how to encrypt your sensitive data and still comply with the schema.

The complete code can be downloaded from: http://keensocial.freeiz.com/blogs/xmlfileformat/xmlfileformat.zip. The project file can only be opened in Visual Studio 2010. If you have other versions of Visual Studio, you have to create a new solution and add the extracted files from the zip file to the solution.

Defining a file format using XML and XML Schema (XSD) in C#/Java - II

The complete implementation of XML file format is divided into the following steps:

1. XML Schema (XSD) definition
• Let an extension for my file type: .xef (XML example file-type)
• Generate an XML file with data compatible to previously defined schema
2. Validate my XML with the schema
3. Compress huge data fragments
4. Encrypt sensitive data
5. Implementation in Java (Compliance with XSD 1.1)

 In the previous post, we have defined an XML schema and an example XML file based on the schema. In today's post, we will read/write/validate the XML file against the schema in C# using Visual Studio.

1. Create a Windows Forms application.
2. Add 3 buttons
• Load
• Load and Validate
• Save

Following is a code snippet for creating an XML document:

string  schemaName = "set.xsd";
XmlDocument document = new XmlDocument();
document.Load(fileName);
XmlElement xmlNode = doc.CreateElement("XmlFileFormat");

XmlAttribute attr = doc.CreateAttribute("xmlns:xsi");
attr.Value = "http://www.w3.org/2001/XMLSchema-instance";
xmlNode.SetAttributeNode(attr);

attr = doc.CreateAttribute("xsi:noNamespaceSchemaLocation");
attr.Value = m_schemaName;
xmlNode.SetAttributeNode(attr);

doc.AppendChild(xmlNode);
doc.Save(fileName);

And code snippet for loading this document:

string  schemaName = "set.xsd";
XmlDocument document = new XmlDocument();  
document.Load(fileName);

The validation is done using XmlSchema class:

string  schemaName = "set.xsd";
string  fileName = "set.xml";

XmlReaderSettings settings = new XmlReaderSettings();
settings.Schemas.Add(string.Empty, schemaName);
settings.ValidationType = ValidationType.Schema;

XmlReader reader = XmlReader.Create(fileName, settings);
XmlDocument document = new XmlDocument();
document.Load(reader);

ValidationEventHandler eventHandler = new ValidationEventHandler(OnValidation);
// the following call to Validate succeeds.
document.Validate(eventHandler);
reader.Close();

If anything is changed in the document we can always call document.Validate() with the OnValidation event handler: 

private void OnValidation(object sender, ValidationEventArgs e)
{
  switch (e.Severity)
  {
  case XmlSeverityType.Error:
    Console.WriteLine(e.Message);
    break;
  case XmlSeverityType.Warning:
    Console.WriteLine(e.Message);
    break;
  }
}

Very frequently we come to problems in XML data when we need to represent large data sets. Due to verbose tags, we need several byte even to write a single data byte. This causes an extreme overhead during serialization and data transfer over slower networks.. There were number of efforts seen in the open source and research communities to overcome this problem. In the next post, we will implement one mechanism in C# to compress large datasets.

The complete code can be downloaded from: http://keensocial.freeiz.com/blogs/xmlfileformat/xmlfileformat.zip. The project file can only be opened in Visual Studio 2010. If you have other versions of Visual Studio, you have to create a new solution and add the extracted files from the zip file to the solution.

Defining a file format using XML and XML Schema (XSD) in C#/Java - I

Occasionally I need a custom file type for new application. I prefer to use OLE/COM Structured Storage since it offers me a compact file-system like hierarchy with Storages and Streams to save application data in binary format. My open source application "UVFSEditor" can view a compound file along with other file formats. The storages are shown in a tree like structure at the left and the streams are displayed in a HexEditor.

For quite a long time Microsoft is not doing anything around OLE/COM structured storage and the company itself is moving away from this file format for its own applications in favor of some XML formats. I have also been looking for an alternative for new applications. XML is a viable alternative since the information can be viewed and edited with simple text editor, validated using XML schema validator, the data can easily be exported to other file formats.

I divided the complete work in the following steps:

1. XML Schema (XSD) definition
• Let an extension for my file type: .xef (XML example file-type)
• Generate an XML file with data compatible to previously defined schema
2. Validate my XML with the schema
3. Compress huge data fragments
4. Encrypt sensitive data
5. Implementation in Java (Compliance with XSD 1.1)

Schema definition
A file type has a definite structure which helps the application to save and load data in a systematic way. An XML schema helps to define the type and size of data along with their relationship with each other which creates a predefined structure of a generated file. XML schema serves the same purpose as a relational schema serves for a database. The World Wide Web Consortium (W3C) has a recommendation for defining XML schema called XML Schema Definition Language (XSD). The most recent recommendation of XSD 1.1 is in candidate status. The language itself is in XML.

Let's start with my sample file-type schema:

<?xml version=“1.0“?>

<xsd:schema xmlns:xsd=“http://www.w3.org/2001/XMLSchema“>
 <xsd:element name=“Document“>
    <xsd:complexType>
     <xsd:sequence>
       <xsd:element name=“Value1“ type=“xsd:integer“></xsd:element>
       <xsd:element name=“Value2“ type=“xsd:string“></xsd:element>
       <xsd:element name=“Value3“ type=“xsd:float“></xsd:element>
       <xsd:element name=“Settings2“>
         <xsd:complexType>
           <xsd:sequence minOccurs=“0“ maxOccurs=“1“>
             <xsd:element name=“ValueX“ type=“xsd:integer“>
             </xsd:element>
             <xsd:element name=“ValueBulks“>
               <xsd:complexType>
                 <xsd:sequence>
                   <xsd:element name=“ValueBulk“ type=“xsd:float“ minOccurs=“0“ maxOccurs=“unbounded“/>
                 </xsd:sequence>
                 <xsd:attribute name=“Compress“ use=“optional“ type=“xsd:boolean“></xsd:attribute>
               </xsd:complexType>
             </xsd:element>
           </xsd:sequence>
         </xsd:complexType>
       </xsd:element>
     </xsd:sequence>
    </xsd:complexType>
 </xsd:element>
</xsd:schema>

Extension for New File Format
XML is a text format, so is XSD. A file in Windows OS almost always has an extension which provides information to the Windows Shell which application should be started once a file of particular type is, for example, double clicked in Windows explorer. For my test application, I take chosen .xef (XML example file-type) as file extension

Sample XML file definition

 10
 Test File Format
 101
 
    30
    
     1.0
     2.0
    
 

I prefer Visual Studio IDE and C# whenever I need to try some demo application which has a user interface. The next post of this blog will include how to read & write an XML file and validate it against our predefined schema using .Net framework classes.

The complete code can be downloaded from: http://keensocial.freeiz.com/blogs/xmlfileformat/xmlfileformat.zip. The project file can only be opened in Visual Studio 2010. If you have other versions of Visual Studio, you have to create a new solution and add the extracted files from the zip file to the solution.

Parabox

Last weekend was lonely and I felt nostalgic for the paranoid game that we used to play during university days. We were all mad for it and spent hours playing it. For few days I was experimenting with HTML5 and CSS3 capabilities in various browsers. I took the opportunities and programmed our beloved paranoid and integrated it as facebook application. It needs chrome, firefox, IE9, safari and opera browsers to run.
The facebook apps is at:
                                      Parabox Game
You can play it here:

Journey to an Unknown Land

Previous 1 2 

You have a feel of safety everywhere. Car drivers don't pass on red signal, pedestrians don't cross the roads here and there. You don't need to be afraid of some pick pocketers or hijackers. Streets are free of dirt. You are relieved from chaos and rush. Until you are in a supermarket, you will see calmness all around.

Life is charming and beautiful. It is cold outside but with room heaters on you don't feel it from inside. You don't need to care for seniors when doing anything. The people are friendly and warm. If you ask for help, most stand by you eagerly. Unlike many friends during my University life, they will not point you to the wrong direction when you ask them for a street or house number.

My company was bankrupt before arriving here. I got a great feel of job safety when a new job was confirmed just the next day after entering into this land. There was a round of walking through the company compound to introduce myself to other colleagues on the first day. Everyone seems very much friendly. The office was located in a nice place nearby to the city. The forest was just beside the office building and the city on the other.

Everything was going perfect. Life is going to settle down in an unknown land without any hassle. It seems I was born in the wrong place and now migrated to my home.

Previous 1 2 

Journey to an Unknown Land

1 2 Next

It all started a decade ago. One of my friends asked me to send a copy of my CV to a company in an oversees country. I did that and accomplished a telephone interview successfully in few days. A journey to an unknown land began in a new year.

Life was not terribly bad in Dhaka as a Software Engineer. The software industry in Bangladesh was in the infant stage at that time and was aiming for boom. My income was distinguishably over the average and daily life was pretty cool with regular Adda among friends. Parents were just 2-3 hours of travelling distance and some family members could be reach in few minutes.

My assignments at the job were very unique experience. Still like others in the developing countries, I was eager to move to some developed countries, partly to achieve more professional challenges and partly to earn more. There are rarely few people in these countries who are not touched by the colorful life of developed world as seen in Hollywood movies.

The journey took place on 14th January, 2010. The first step in the unknown land on January 15 was a shock. The minus temperate was an unbelievable experience in the life. The 200 km trip by car was terribly beautiful. The highway looked like a dream and the roadside cities were organized and  disciplined comparable to those  only found in fairytale.

It was a dream come true!!!

Open Source Projects

Following is a list of open source projects that I initiated and programmed:

1. Universal Virtual FS Editor
2. Flexible Registration
3. JavaScript 2D/3D chart
4. Syntax Highlighter (YASH)
5. Salat Times (Firefox add-on)
6. C++/Java/C#
7. Keen Social (discontinued)

Developing Firefox Extensions

1. Create a development profile: 
   
   
   
   You need to install several extensions which may slow down firefox performance. So create a separate development profile using the following steps-
   

   • Open command prompt
   • [Firefox Installation Directory]\firefox.exe -no-remote -P
   • Create a new profile.
   • Start firefox next time with the command: [Firefox Installation Directory]\firefox.exe -no-remote -P [new profile name]

   
   
2. Install some useful extensions: Firebug is the most useful one.
   
   
   
   • Firebug: https://addons.mozilla.org/en-US/firefox/addon/1843/
   • DOM Inspector: https://addons.mozilla.org/en-US/firefox/addon/6622/
   • Extension Developer: https://addons.mozilla.org/en-US/firefox/addon/7434/
3. Set some developer preferences: Type about:config in the address bar. Set or add, if do not already exist, the following properties:
   
   
   
   • javascript.options.showInConsole = true.
   • nglayout.debug.disable_xul_cache = true.
   • browser.dom.window.dump.enabled = true.
   • javascript.options.strict = true.
   • extensions.logging.enabled = true.
   • dom.report_all_js_exceptions = true.

   
   
   
4. Directory structure: Different from usual programming frameworks, Firefox extension has a predefined directory structure for files. The browser will look in specified folder when it needs a file for e.g. installation, preference or visual overlays.
   
   
   
   • For new extension: Use the wizard in http://ted.mielczarek.org/code/mozilla/extensionwiz/. Copy the contents of the generated zip file into [Profile]\extensions\] folder. It is a folder with the name of your desired extension name. Rename this folder name into the extension ID from install.rdf or use some unique name like an e-mail address. Restart firefox. Your add-on should appear in the browser. Check Tools->Add-ons if your extension is listed there.
   • Existing extension: Copy extension into the [Profile]\extensions\] folder. Remember the filename or sub-folder for the extension must be same as the extension ID in install.rdf or a unique name like an e-mail address.

How to contribute to Salattimes firefox extension?

Analogous to my previous post this post describes how to check-in/check-out the firefox add-on project salattimes from google code server.

1. Check-out: You need a subversion client to check-in/check-out code from the SubVersion source control server. Windows users should download TortoiseSVN from http://tortoisesvn.net/downloads. Linux and MacOS users should google to find the appropriate client.
   
   Windows users just right-click on explorer in your development directory and then click "SVN Checkout...". The following dialog will appear.






If you click "OK", a dialog box will appear to provide username/password. You can now start programming.

Non-windows users might need the following command on the console:
svn co https://uvfseditor.svn.sourceforge.net/svnroot/uvfseditor uvfseditor

The rest is same as the previous post.

How to contribute to UVFSEditor?

Last few days, several programmers are accepted to join the development efforts of UVFSEditor. In this post, I will briefly write the steps necessary to involve in the development.

1. Check-out: You need a subversion client to check-in/check-out code from the SubVersion source control server. Windows users should download TortoiseSVN from http://tortoisesvn.net/downloads. Linux and MacOS users should google to find the appropriate client.
   
   Windows users just right-click on explorer in your development directory and then click "SVN Checkout...". The following dialog will appear.






If you click "OK", a dialog box will appear to provide username/password. You can now start programming.

Non-windows users might need the following command on the console:












svn co https://uvfseditor.svn.sourceforge.net/svnroot/uvfseditor uvfseditor


2. Check-in/Commit:




First of all, be careful, don't check-in until you are absolutely sure what you are checking-in does not break the existing code.

Right click the file or folder you want to check-in. The following context menu should appear:



First click "SVN Update", then click right mouse button again and click "SVN Commit...". The following dialog should show up:



Write an appropriate message on the "Message:" box and click ok.




You must write a message describing the changes you have made.

Now you are ready for UVFSEditor development!!!

3. Development: You need Visual Studio 2008 to develop (e.g. you can also use MonoDevelop), but the UVFSEditor will run only under Windows OS. Open the UVFSEditor.sln solution and compile. To understand the code hierarchy, first look at "UVirtualFileSystem" project. You will find there classes which resemble classes in System.IO package. The classes in "DotNetFileSystem" and "OleStructuredFileSystem" are inherited from those in "UVirtualFileSystem". The "UniversalHexEditorLib" implements a extremely flexible hex viewer as a .Net control. "UVFSEditor" is the main application project which implements a GUI and glue all libraries together.
   
   The "uvfsEditorSetup" projects creates an MSI installation package for the end users.
4. Forum:There is a development forum under the following URL: https://sourceforge.net/projects/uvfseditor/forums/forum/1142984. You need to log on to access this forum. You should communicate with each other over this forum.

Too much to learn for Programming? - Execution environment

There are many popular languages for programming and to learn only a subset of them from scratch is a nightmare. The fact that each language needs some tuning on each platforms makes it more complex. Therefore, programming concepts should be learnt first on an abstract level and adapt those concepts to specific platform when need. A programmer should have fair level of concepts on the following areas to master any programming language as well as leverage knowledge transfer to other programming languages:

1. Execution environment
2. Memory Architecture and Allocation/Deallocation
3. Object Model
4. Algorithms, Complexity and Data Structure
5. Language Syntax
6. Compilation & Linking
7. Packaging and Deployment

Next few days, I will briefly explain each of these areas and today start with the first one – execution environment.

EXECUTION ENVIRONMENT

Execution environment (EE) is a layer over the hardware on top of which your programs run. It is very often the operating system, but could also be other platforms like virtual machine (JVM or CLR) or even browser (e.g JavaScript) or other applications (e.g. VBA on office application). Your program must have one or more target platforms. Software targeting for one platform may not run on a different platform.

There are various constrains of EE e.g. OS version (32/64-bit) compatibility, requirement of certain run-time libraries etc, some supporting libraries. Some software are OS agnostic, but depends on some other execution environment layer over OS. Java and .Net programs are the best examples of such runtime environment. It is important to understand the inner workings of an execution environment if you want to be expert programmer. How EE understands your file, loads it and what information your executable file must provide so that EE is able to load it.

An example will make it clear. Windows OS considers a file executable if it has some specific extensions (e.g. .exe, .bat, .com etc.). But when the OS starts to load the file, it looks if the file has certain information at predefined location to set up the correct execution environment (DOS, Win32/64, WOW64 ). The .exe executable file is mostly of the portable file (PE) format which is derived from Common Object File Format (COFF). Let's examine the steps in brief that windows OS pass through to load an executable file:

1. Validate parameters and flags that the shell (or programmatic CreateProcess call) specifies.
2. Open image file (.exe).
3. Create a process.
4. Create initial thread.
5. Perform other initialization.
6. Execute initial thread.
7. Load required DLLs and start execution of initial thread.
8. Eventually the entry point function (main/wmain/_tmain for console and WinMain for GUI application) is executed.

So this is a fair amount of work for the OS to load and run an executable file.

The process is much different for a so called platform independent programs. During compilation a .Net compatible compiler adds reference to the MSCorEE.dll which is referred as Microsoft .NET Runtime Execution Engine to the generated PE or PE+ (for 64-bit) executable file. The steps for loading and running a .Net image are roughly described below:

1. 1..6 of the native image load described above.
2. Loads MSCorEE.dll in the process address space.
3. The initial threads starts executing calling a method in MSCorEE.dll which initializes CLR, loads EXE assembly and then calls its entry point method.

The process is more straight forward for scripting code. The host application is responsible for providing memory and object model to the script code. The capability of script programs is also more restrictive than native and VM programs. For JavaScript code, the browser has a JavaScript engine which parses, loads, and executes the code.