Sentiment Analysis of Transliterated Texts in Hindi and Marathi Languages

This blog is about the presentation I made for my dissertation in the NLP – ML domain. Sentiment Analysis is easy for English. It’s a little but not much difficult for other languages such as Hindi or Marathi. It gets difficult when the script used for the other languages is Roman. What makes it most difficult is when that transliterated text is a mixture of multiple languages. In our case we are considering two combinations of English-Hindi and English-Marathi. This mixture of languages is called code-mix. Usually one of the language (usually Roman script) is used for the textual representation. How to analyse the sentiments in this situation.

Ordinary Sentiment Analysis

The steps involved in the usual analysis is as follows:

  1. Normalize the word tokens
  2. Perform the POS Tag analysis to get the words tagged as either noun, adjective, adverb and verb.
  3. Use wordnet to get the identifier and use senti-wordnet to get the sentiment associated with those words along with the corresponding tags we found in the POS tag analysis
  4. Convert the words in to features:
    1. Remove stop words
    2. Remove the words senti-scores that are not important such as everything except adjective and adverbs, since they are the most important ones where sentiments are concerned
    3. Create an ngram feature to hook words with its context
  5. Feed it to a classifier for training and then test on the subject to calculate accuracy
    1. Cross validation is used to not over train
    2. A threshold is found that balances overtraining and generality to analyse unseen data

Problems with the generic approach for mixed code

Although the steps are more involving and I’ve missed a few, but the general approach is covered in the above steps. The only problem with the transliterated mixed-code is that it doesn’t work with the above steps. Why? Because the transliterated text does not have a fixed grammar, nor the words are spelled correctly. The sentence construction does not follow a single language specific rule. If the chunks of each language are taken out, then in many cases; they don’t form the correct grammatical syntax of their respective languages. The last part happens due to the mixture of language rules. An example is in order: Maine tujhe Good luck tere exams ke liye nahi bola because tumse na hoga. In the previous statement, the usage of English is following the contextual grammar of Hindi language and hence their POS tagging and their associated sentiments are highly ambiguous in this case. This leads to the conclusion that mix-code scripts require a different approach for this purpose and that is exactly what my dissertation is all about.

Proposed Approach

Since figures speak the thousand words, I’d let the following figures illustrate the approach that I took for the purpose of sentiment analysis.

The above figure gives all the important steps that are needed for this approach as listed below again, for those not interested in spending time on the figure.

  • See if the word tokens are an emoticon or a known slang like lol, etc and then replace them with appropriate language works like 😊 will become smiling and lol will become laughing out loud.
  • Transliterate the remain words in Hindi/Marathi language script (Devanagari) and look up the words in Hindi/Marathi dictionary for its existence. Check for spelling variations. If found tag them as such.
  • Lookup all the words in English dictionary and tag them as such. If there is tie with the other language then use word frequency probability to break the tie.
  • Loop up the sentiment scores from sentiwordnet of each languages (Hindi/English), for each token based on their language tags. Except in case of Marathi, which presently does not have the sentiwordnet, in that case, use bilingual dictionary to get the meanings of Marathi words from hindi and English and then follow the process for those languages and merge the scores.
  • Finally convert the words to features using ngrams with their corresponding sentiment scores and perform the cross – validation of training and test data to see the accuracy of results.

These steps a lot more complicated to implement and after using python Scikit-learn to implement these steps I was able to get the result as shown in the following graphs.

Hindi Language chart of max accuracy obtained
Marathi language chart of the max accuracy obtained


The worst part is, even these do not work as good as sentiment analysis for a single language works. The best I could do is get above 50% accuracy for Hindi and English  and 80% accuracy for Marathi and English combination. Now 80% seems like good but there is a caveat. The problem with Marathi / English combination is that the data is general skewed by the presence of fixed words. The data I used came from youtube comments and there were some words in both the positive as well as negative groups, which tend to be the absolute indicator of the sentiment polarity. This makes the system works but in truth it really isn’t working. I’m not proud of the results, but I’m glad I learned a lot from this. Until next time.


Wiki.js – The best wiki software ever

I have spent a long time trying to find the best open source software to collect the knowledge that I have gained, for posterity.  However, after trying multiple available options, such as Tiddly Wiki, Media Wiki, etc. I settled with Wiki.js. I’m not claiming that there isn’t a better solution out there, there must be. My major point of satisfaction with this tool is that it is exactly as I expected my knowledge base to look like.

My reason for using a wiki is to be able to keep a single place for all my study notes, command and code snippets, that I can search, collaborate with others on and publish. It is a knowledge tracker that I can go back to again and again and update or revise the knowledge that I have accumulated. This requirement is easily solved by many note-taking systems such as One Note, Evernote and even Google Drive. The major reason why I didn’t go with those solutions are very specific to each of those systems. One note requires Microsoft machines, whereas I usually work with Ubuntu. Evernote isn’t as great as it was in free mode and Google drive requires a Gmail account for all the collaborators. I needed something that supported Markdown to edit my pages and be able to call it’s data my own, so that even when the service goes down, I don’t loose data.

Let’s look at why Wiki.js works for me. I have hosted wiki.js on Google compute engine (I have 12 months of free trial, so why not!) and the only requirements it has is git, mongodb and nodejs. I work with these technology on almost daily basis, so setting up was a breeze. The most interesting aspect of this system is that it keeps the textual content as .md files on the file system that is a git repository and also allows us to push to a remote repository. This is great on so many levels, since it not only backs up my data automatically to my favourite git repository service provider (bitbucket in this case – since it gives private repos for free), but also can be edited by updating the repos separately via any editor and pushing the changes across to wiki.js. This way, the task of writing the wiki becomes much easier and can be done on any editor. I’m really digging the idea of using git as the backend for versioned storage of files, especially the text files.

There are many other benefits, such as easy of use, administration is intuitive and simplicity is the core of its design. However; there are some cases where wiki.js is rough around the edges, such as the search function is not exactly performing well and neither was I able to get my image uploads to work properly. Although, I do have  a work around for many of such issues, I would love to see wiki.js grow up to be the awesome wiki software that it promises to be.



Home page for wiki.js:

Github page for wikis.js :


Async Core – multiprocessing and asyncio based solution for backend services

The why?

Although, there have been great products that existing since a long time, which does exactly and more than what this project is trying to accomplish, namely – Celery. However, the basic premise of this effort is again based on a very simple idea, which happens to be the mainstay of python community and that is – KISS. Most of the time, a project begins humbly with small code based and small set of functionality and over time, tries to do a lot of things and while doing so, ends up accumulating huge sets of features that are overwhelming for a beginners to start with. Therefore, keeping the same problem in focus, I decided to make a celery like project which will do less instead of more. It will need less expertise with python (yes, the project requires python) for creating heavy backend services, by focussing on writing those backend services instead of the infrastructural code to run those services.

What it supports?

As of now, the project is aimed at using the usual suspects of existing middleware technologies such as rabbitmq, redis, zmq and kafka. As of writing this blog, rabbitmq and redis are supported and work is being done for the rest two. It is written using asyncio module in python 3.6 and that is the version of python recommended to use it. In coming sections we will see, how this project does enough with less requirements. As soon as zmq is supported, the need for another middleware system, like in the case of other three, will cease to exist. More on that later.

How to get it?

The source code for asyncexec is available on and it is released under MIT license, as it seems to be a very open one and without much hassles. One can go through the README to install and use the project and following is the what it says, in order to install it.


  1. Create a virtual environment:
    $> virtualenv venv -p /usr/bin/python3.6
  2. Activate the virtual
    $> source venv/bin/activate
  3. Use pip install to get the module and also install all the required modules it has as dependencies
  4. pip install git+

How to use it?

The repository contains two examples to show how to use the system, under the asyncexec/examples folder.

  1. simple_queue_based; shows how to use a simple rabbitmq based queueing system to run handlers on sending the events as rabbitmq messages
  2. workflows: expands on the previous example and shows how to break down tasks as multiple commands and use concurrency to achieve high performance using parallel group commands or create a workflow by combining it with sequential group commands.

Instead of describing the whole example, I would only like to point the attention to a very simple way of using the system, which is shown below.

#import the class
 from asyncexec.exec import  AsyncExecutor
 #Create the object and give configuration in constructor
 async_executor = AsyncExecutor({
     "rabbitmq": [
         ['pyasync_core_request', 'pyasync_core_result']
 # Define handlers
 def test_method(data):
     print("Hello", data)
     return str(data) + ': response'
 # Start the process
 # Example message to send
 # {“event”: “test_method”, “data”: …}

That is it. The test_method handler will be added as an observer (Observer Pattern) and the listener will call the observer to handle the event they are listening on. This mapping between handler and the event is done by sending the event as a message, where the key “event” would be the name of the handler method and the data key will provide the handler method with a single argument input to use for its execution.

Coming soon

Although the main idea is to keep it small and simple for handling backend services, it is still lacking and is currently too simplistic. Although not fit for production use at the moment, very soon it will be. The following are the major upcoming features that I intend to include in this project:

  • Kafka and ZMQ support
  • Queue based handler distinction to decide which instance can run which all handlers
  • Multiprocessing improvement to enable the user to decide which handler should be run in a separate process/thread.

These are the few urgent things that will be worked on in near future, however; I’m open to inputs from all sides and would love any kind of contribution to make this a success. Until next time.

The Human and The Divine

This is a small post to express a thought that I have been forming for a few days now and it is important for me to express it, in order to internalize it. As a muslim, it is a common thing to hear and say Allah-hu-Akbar (Allah is greatest) or Subhan Allah (glory to God) or Alhumdullillah (all praises are for Allah). However, I always wanted to understand for myself what do they truly mean. Their meaning, not lost in translation, but lost in shallowness of my own understanding. This is not a post about Islam, nor is it a post about the God, who we have created in order to understand Him. This is about the absolute infinity that Georg Cantor introduced in mathematics, the Absolute that Newton was unable to find due to his inertial frames, the Reality that mystics experience and the concept that religious are pretending to worship. The source of all existence, the power that runs everything and the owner of words behind the Big Bang.

All the religions and the mystical experiences have in common the basic fact that God is beyond comprehension. The question I had for myself is that, how much of my comprehension have I exhausted in order to try and comprehend. It is very natural to give up on understanding something that you just can’t understand. But the question is that off the effort and how much of it was poured before giving up. What was the nature of surrender with which the subject matter was given up? Was it that it does not matter so let us move on to something more close to our desires? Or was it that I have invested my life, witnessed what was out there, exhausted my emotions, challenged my intellect and yet I do not comprehend! The last type of surrender is what we see mystics perform. What is gained by that kind of physical, moral, intellectual, emotional exhaustion and how worthy is that endeavor?


We have not but one life, a mystical existence which may as well be mere simulation. The physics coded in by programmers like in a matrix. Or maybe, the universe truly exists, in its vastness and greatness. It is infinite and not in the sense of number line, but in our exploration of it. In this one life, small existence, we can find purpose in loving who are close to us, doing what we must to protect and provide for ourselves and that would still be worth it. However, how much greater is the worth to stand at the edge of this world and gaze out in the universe to seek the face of the one who created it. Or look down on the ground and into the center of earth, inside the smallest particles to the force that holds it all together and see it as an extension of the words that created it all. Science is secular and rightly so, to avoid to superstitions from running amok and impeding the progress of knowledge. But, only the method is secular, not necessarily the intention. Newton gave the science it’s one of the biggest boost and yet his intention was to come face to face with God. So was Descartes, who busied himself unraveling the structure of the creation. Why cannot a man seek God through science? Since it is but an extension of philosophy? Philosophy was merely speculative and deductive as long as Aristotle was the fashioner of its process. It become empirical and observation based over time, but does not mean, speculations are not happening. what is a hypothesis if not a speculation first that are backed by observations later?

How deeply the connection runs between what is reality and what is symbolic knowledge like mathematics? As a human, the thought that there is such a connection at all and that a human mind was able to comprehend such a connection is a case for God and divine purpose of humanity. This brings me to the point I mentioned at the beginning of the post, that I wanted to understand what it means to say God is greatest. The fact that we are so insignificantly small that billions of people are living on a planet, on the lands which are 1/3 of the surface area of that planet. A planet so small compared to other planets in the same solar system and a sun so huge that none of the planets comes close in comparison. And yet, the sun, a start, which is less than the average size of existing stars. The billions of such stars in the universe! How insignificant it can get for human! There cannot be a shred of arrogance in a person, when they realize how small and helpless they are on a distant planet in the backyard of an average galaxy, which is in the trash can of universe. And yet, these humans were able to comprehend this. The fact that humans have embarked on this journey of understanding due the capability they are endowed with in terms of intellect, becomes a case for a God who cares enough to give that intellect. Therefore, it is an amazement to me, that the human beings, being as insignificant as they are, are given the purpose so magnificent that such a purpose cannot be denied. A moral obligation, a divine purpose.

Photo Credit: Joan Larsen, Wikipedia

Dreams in the machine


When I-Robot movie came out, it really put me in a bad place. Being young and a believer in a uniqueness of humanity, it was very difficult thing to come to grasp with a position, where a man-made intelligence acts with free will and not only that, it also dreams. I thought, the greatness of man was that he is intelligent and the idea that something more intelligent can be created by our own hands seemed like an impossibility at the time. Now, I’m not only looking forward to it, but devising plans for it to become reality. What fascinated me with the idea of dreams of a machine is the fact that we ascribe that ability to be quintessentially human. Although, I do believe that animals must be dreaming too, as suggested by an article on national geographic here. Which is why, I thought it would make sense if an AI starts dreaming too. It seems not only far fetched, but also useless to have this kind of conversation, unless we understand why we dream. Now there are many theories why humans dreams, but I am inclined to the idea of dreams being a classroom training for our mind. Here is what I think happens when a person dreams: The brain creates a simulation of real world in our head, disconnects our body from the actions we take in dreams and uses it to send feedbacks inside the dream. Our mind is incapable (most of the time) of understanding that it is a dream and deals with it as if the events are happening for real. Most of the time the dreams are very exciting and our responses to the events happening in the dream are also extremely intense, as in the case of nightmares. This process allows our mind to go through a particular experience that it hasn’t gone through before or may have to go through in the future. The past too is repeated and mostly, with variations. If we think about it, we can see that this is how the brain teaches itself to handle different situations that it needs to prepare for. There must be a dream generator function in our head that creates a scenario which is run through in our dreams and our mind learns to deal with that situation. So, it is a kind of a learning system, that teaches our brain to explore the possibilities outside our everyday experience.

What has this got to do with machines?

I believe machines learn via dreams too, only we do not call it such. We call the simulations. The designer of simulation creates a scenario and the machine is run through that scenario, which allows us to train the machine for different eventualities without having to incur the cost of training the real thing in the real world. In the world of machine learning, simulations happen with training of the algorithms with data. This training data is collected from the real world and then passed to the algorithms, which will generate the world-model based on the data and then we can test the algorithm with another set of test data for its accuracy. These systems are not absolutely accurate and the failure rate goes down when the training is properly balanced between under and over training. Even in case of reinforcement learning systems, the system learns by rewards and punishments based on the moves it makes by either exploiting the known best path or by exploring to figure out even better paths and updating its parameters or world-model. It is also understood and accepted that, the data keeps changing and therefore, there is a need to constantly learn and use some feedback mechanism to learn continuously. This is very much similar to way organic beings like us learn too. However, in the above approach to continuous learning, there is a limitation that comes due to the feedbacks being generated from the data already seen. Wouldn’t it be interesting for machines to also try some data that it has not seen? Much like the way we dream, where we are jumping off of a cliff and flying or walking in pyjamas/naked in office or school?

Dreaming away to glory!

The proposal I’m making with this post is to design the learning components in such a way that there would be two phase of its working:

  1. Production mode: Production mode is normal operations where the machine is given the real world data and its output is consumed by the real world. The feedback here generated would be directly fed in to the system while its online.
  2. Dream/Simulation mode: In this mode, there is a need for world generator engine (world here represents the environment of the machine [function/algorithm]) with which the machine is suppose to interact. The world generator is what simulators today work as, however, with the slight change that it is suppose to come up with new and interesting scenarios that may or may not occur generally in real world. This would allow the algorithm to learn something it has never seen before and would not see possibly for a long time.

The importance of simulator / world generator

The biggest takeaway from this idea is the concept of world generator to generate simulation scenarios. To understand it, imagine that a game automatically creates the levels of the game by studying and understanding the existing levels (meeting with all the constraints such as level must have a way to finish successfully and failure should be based on the scale of difficulty). This ability to generate levels in game would cause the games to take exponentially long time to end and possibly make them very boring. However, for a machine, this routine game play with tiny to huge amount of variations will end up making the machine intelligence very strong. A good use case in computer science, which may speed up the idea of such generators, is the concept of generating test cases. There are existing frameworks and service providers that allow randomizing the data for testing, to cover the corner cases while testing a given service. These can be taken further by allowing the generator to study the existing cases and coming up with many new variations of the existing test cases, while ensuring consistency with the constraints of the system. This will not only allow the developers to capture the bugs early, but also free up a lot of time developers spend writing those test cases.

In conclusion, this blog post is just a way to throw the idea out there and develop it over time. I hope I will be able to inspire a new startup with this post. May be, I might be the one to start it up! It would be fun to have machines with the dreams and then another sci-fi movie like I-Robot will leave the genre of fantasy and enter reality. Cheers!

Greek mythology and traces of Abrahamic religions

Based on the works of Robert Bowie, Jr’s.  Article here:


This post is more of a summary and a place to capture the questions associated with the above link, which is a post that talks about how the greek mythology and early history/myth of abrahamic religions are actually a narration of a set of events from polar opposite perspectives. It shows how the Zeus/Hera characters (the first man and women couple from greek mythology) are the same as Adam/Eve (the first man and women in Abrahamic lores). The main aspect of this article is to stress the concept that both the greek mythos and abrahamic lores are inverse and averse to each other’s view. In this post, we will look at those points and mark the points of conflict as well as the promising area that needs further research.

The greek people had a rich mythology and were made famous by movies like Troy and Hercules for a laymen like me. It’s richness, epics and great parallels to human condition we face on day to day basis is a topic of its own, however; this post is about something very specific,, where the relation between greek mythologies and stories from Abrahamic religions is being considered to find a possible connection. Since, Greeks never believed in Creator God, their mythology has three level of hierarchy starting with Uranus and Gaia to Kronus and Rhea to Zeus and Hera. The basic idea of Greek mythology as suggested by Robert Bowie, Jr. is that it is centered on the idea of getting away from the God of old testament. It views the narrative of Greek mythology as Zeus/Hera couple escaping from the yoke of Old testament by eating the apples from the Garden of Hesperides (Eden) and gain the serpent’s enlightenment (Knowledge of Good and Evil as promised by satan).

Garden of Hesperides and Eden

As can be seen in the image, the Garden of Hesperides seems to be a very interesting place. What is more interesting is the fact that it belongs to Zeus and its apple were wedding gifts by Hera. Somehow, Athena ends up owning this gardent and the nymphs seen in the image are daughters of Atlas. What is interesting to see, is that there happens to be a tree with entwined serpent, which is reminiscent of Garden of Eden with Serpent in it. Also interesting thing to note is that both stories have apples associated with them. As, Robert Bowie, Jr. suggests, it can be seen that serpent does play a prominent role in Greek mythology as enlightening being. Following are some of the other images to drive that point home.


It is also visible near the feet of Athena in the picture to the left and it is usually shown as rising towards upwards. This kind of suggests the possibility of serpents achieving importance. Robert Dowie, Jr. suggests that serpent worship was prevalent in ancient Greece. Serpent worship forms an important aspect of all the ancient polytheistic religions and therefore this imagery cannot be discounted.

Hercules had to steal apples from Garden of Hesperides and had to defeat Nereus, who was the ruler of the sea and was replaced by Poseidon. Nereus knew where the location of the garden was and according to the Robert Dowie, Jr. this Nereus character is Noah, who survives the flood. Even the fact that Hercules fights and wins against Kerberos, is the idea of defeating the three sons of Noah post flood. We will return to post flood scenario. Another interesting thing that happens during the time when Hercules tries to steal the apple, he uses help from Atlas, whose daughters are protecting those apples. Atlas agrees to help, since he wants to let off the burden of the sky and give it Hercules. Hercules tricks him to get the apples for him and then carry the burden of sky again, before leaving with the apples. I mentioned this story, because Robert Bowie, Jr. suggests that the act of holding the sky above is more like moving the heaves (old testament’s God) away from the lives of people where they can live with liberty and freedom.

Abel/Seth and Cain – Ares and Hephaestus

According to old testament, Cain killed Abel because he was jealous of him and was cursed to wander the land, with some kind of mark to identify and protect him from would be assailants. I still wonder who those assailants are being referred to, if this is the beginning of humanity on earth. However, Cain disobeys is curse to wander and decides to build a city. His children are famous for metal works, music and sciences. On the other hand, Seth takes the righteous place of Abel and settles with a pastoral lifestyle and lives with a nomadic way of life. It could be that these nomads were rough, unruly in their demeanor and would be willing to get in to fight easily due to their simple way of life.

Hephaestus was the favored son of Zeus and Hera and was famous for his metal works. Just like Cain’s children were and Cain himself fashioned a weapon out of his plough. Ares, inversely is the war oriented God of Greeks, who is despised by the entirety of Mount Olympus. Except for the two for two relation, there isn’t much that seems very striking here and so we would move on.

The Flood

According to old testament, Noah with his family along with animals on ark survives the flood and the world is inhabited by the offspring of Noah. But over time, the evil before the flood starts creeping back to corrupt the offspring of the three sons of Noah and in time, the whole evil thing becomes prevalent and gets mainstream with Nimrod making it to big time. Nimrod is compared to Hercules and the 12 labours, which includes getting the apples from the garden of Hesperides, is considered to be the enactment of resurrecting the evils ways of the past. It is interesting to note, Nimrod, also called the mighty hunter before the lord, is compared to Orion constellation. This constellation is also example of another dying god Osiris, in Egyptian myths. The other association of Nimrod is with Baal, the father of Tummuz. It forms a trinity of sorts, with Baal (dying god) the father, Semiramis as wife (and daugther in some references), Tummuz as son. Baal was also known as Hubul in the region of Arabia, whose worship was prevalent in Arabia during the time of Prophet Muhammed. We will be looking for more clues as to who Baal is and how he is relevant in following posts.

The question that keeps coming back to me, is that the whole premise of Rober Bowie, Jr. is invalid since the whole Zeus and Hera being the first couple doesnt make sense, especially since they both are children of Kronous and Rhea, who inturn are children of Uranus and Gaia. Another important aspect missing is, what about the brothers of Zeus: Hades and Poseidon. Where do they fit and what about the titans? Who were they? These are the questions that I would like to find the answers for and will explore in the upcoming posts.

Python Asyncio + cython + lua

To begin, I would first like to answer the obvious question: Why would anyone want to use lua when already using python? The answer is simple, legacy code. This experiment arose from the evaluation of possible candidates to replace an existing c/c++ system with limited throughput in terms of request per second with something like python or erlang. In this post, we will be focussing on python, since it was an experiment borne out of my own curiosity. As for the usage of lua, let’s say there was a lot of business logic already written in it and the requirement is to reuse that code base. I chose to use LuaJIT, because it already proved to be fast enough for our requirements. So, the next obvious step was to make my python async code talk to lua. Of course, there is more than one way to accomplish that but what struck to me as an awesome option was to use Cython for this purpose, since it keeps a lot of my code still written in python and generates the necessary c code to communicate with luajit. We will go through step by step assessment of this combination to see what works best. The source code for this blog is here.
Requirements and set up
1. Python 3.5
2. Cython
3. ZMQ
4. LuaJIT
First create a virtual environment for this experiment using the following steps to keep the rest of your python setup safe from this experiment. Install virtual environment, if it is not already installed and then go ahead and create the virtualenv (replacing the path for python as per the system you are using) and activate the virtualenv.

$> apt get install virtualenv 
$> virtualenv venv -p /usr /bin/python3.5 
$> source venv/bin/activate

Once the virtualenv is installed, we will go ahead and install all the packages that we will require for this exercise.

apt-get install lua5.1 liblua5.1-dev luajit-5.1
pip install cython zmq aiozmq msgpack-python uvloop

Let’s setup LUA to do something, not necessarily interesting. The following lua code simply does some random calculations and return the random value.

 function getRandomValue() 
   print('Caling Random')
   x = math.random()
   x = math.floor(x * 10) 
   x = x + 1

print('Priming Run')
 address = ''
 port = 67

As can be seen, the line: print(‘Priming Run’) is there to show that the lua when loaded from c port, does the priming once. If you run this code using the following command, then it gives the output as shown below:

$ lua lua_code.lua
Priming Run

It is time to write some C code to run this lua file. Notice the main function at the end, which we will remove later to be able to call to this file from python.

/* lua_runner.c */
 #include <lauxlib.h>

/* Convenience stuff */
 static void close_state(lua_State **L) { lua_close(*L); }
 #define cleanup(x) __attribute__((cleanup(x)))
 #define auto_lclose cleanup(close_state)

static int on_recv(lua_State *L, char *buf, size_t len)
   lua_getglobal(L, "getRandomValue");
   lua_pushlstring(L, buf, len); /* Binary strings are okay */
   int ret = lua_pcall(L, 1, 1, 0); /* 1 argument, 1 result */
   printf("ret: %d, buflen: %ldn", ret, lua_tointeger(L, -1));
   lua_pop(L, 1);
   return ret;

int myfunc()
   /* Create VM state */
   auto_lclose lua_State *L = luaL_newstate();
   if (!L)
     return 1;
   luaL_openlibs(L); /* Open standard libraries */
   /* Load config file */
   luaL_loadfile(L, "lua_code.lua"); /* (1) */
   int ret = lua_pcall(L, 0, 0, 0);
   if (ret != 0) {
     fprintf(stderr, "%sn", lua_tostring(L, -1));
     return 1;
   /* Read out config */
   lua_getglobal(L, "address"); /* (2) */
   lua_getglobal(L, "port");
   printf("address: %s, port: %ldn", /* (3) */
   lua_tostring(L, -2), lua_tointeger(L, -1));
   lua_settop(L, 0); /* (4) */
   char buf[512] = { 0x05, 'h', 'e', 'l', 'l', 'o' };
   return on_recv(L, buf, sizeof(buf));

void main()

To run this code, we need to first compile this code. Run the following command to compile this the lua_runner.c code, which will result in a.out file that can be run to see the output as shown below.

$ cc -w lua_runner.c -I/usr/include/luajit-2.0 -lluajit-5.1 -lm -ldl
$ ./a.out
Priming Run
address:, port: 67
Caling Random
ret: 0, buflen: 10

The lua script gets loaded by the first lua_pcall function call, which prints “Priming Run” output and then the rest of the code runs to by calling the getRandomValue function in the lua code. Up to this point, what we have done is to simply call from C code to lua. Now we will Call this lua method from Python using Cython.

Python -> Cython -> Lua Code

To start using cython, we must write python a little differently than we are used to. Cython allows python code to be converted to C code and be compiled as a loadable library. This library can be access from normal interpreted python (cpython) as if it was calling some other python module. There are a lot of benefits of using cython, which I would not be going in details of. However, to learn more about Cython, you can follow this link to its documentation. We need to start by removing the main function from the lua_runner.c file, creating a header file with declaration for myfunc() function and then compiling it to a library and writing the cython code as show below.

 /* lua_runner.h */
 #ifndef __MYCODE_H__
 #define __MYCODE_H__
 extern int myfunc();
 #endif //__MYCODE_H__

The lua_runner.c does not need the main function anymore, since we want to compile it to .o object file. The header file is used by the cython file (pyx) shown below to access our library for myfunc function.

# lua_runner_cython.pyx
 cdef extern from "lua_runner.h":
   cdef int myfunc()

def callCfunc():
  return myfunc()

Run the following commands to compile the cython file to c file and then all the c files to object files and then create a shared object file from the collection of object files.

$cython lua_runner_cython.pyx
$gcc -g -O2 -fpic -c lua_runner.c -o lua_runner.o -I/usr/include/luajit-2.0 -lluajit-5.1 -lm -ldl
$gcc -g -O2 -fpic -c lua_runner_cython.c -o lua_runner_cython.o `python-config --cflags`
$ gcc -shared -o lua_runner.o lua_runner_cython.o `python-config --libs` -I/usr/include/luajit-2.0 -lluajit-5.1 -lm -ldl
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from lua_runner_cython import callCfunc
>>> callCfunc()
Priming Run
address:, port: 67
Caling Random
ret: 0, buflen: 5

As can be seen, we are able to import the lua_runner_cython module from the python repl and also able to see the results of the lua_code.lua file when callCfunc() is called. We have setup the python to lua calling. It is time to make a async server which will forward the calls to lua client, print these results and return a value. First, we will directly try to call the lua code from our async server and see how much performance we can pull out of such a system. I am currently running the code on a vagrant box and the resultant performance is not representation of true performance that can be achieved, nevertheless, we will be able to identify and map the difference in performance by running the varying approaches to this problem on the same machine, to get the idea of which approach is most appropriate for application such as this.
Next step in the line would most obviously be able to call this function from python server, where this call will represent a call to backend service. However, I would prefer to not go the route of calling this method from a threaded server, since it will require locking the call so only one thread can access it at a time. It is also a problem because the calling a c port call from thread on GIL would not gain anything in terms of concurrency. Therefore, it makes a lot of sense to me to use queueing between the threads that want to call the Cython function and the thread which will do the actual calling. If we can maintain a proper rpc style wait while calling, then we will not have to worry about locks in our code. Although, this approach removes the necessity for the developer to use any locking, it simply abstracts the locking behaviour inside the queue data-structure, which is not free from the problems present due to GIL. Therefore, we will separate the processes in to two, where 1 threaded server will make calls to the Cython function by putting messages on the queue and one or more worker processes will respond to the call. Since, the queue is involved, the call becomes async and therefore, it requires our special attention. We need to make sure that the returned messages on return queue are forwarded to appropriate thread waiting on response. We can accomplish that by maintaining a correlation id to keep track of request and response. However, in order to lookup the correct thread by correlation id, we will have to maintain a separate data-structure (dictionary) to manage that. Instead, we can simply use event queues, where the appropriate thread will be picked up by event poll as and when messages arrive. Let’s see how that works and if at all that works.
Python Async IO calls LUA from cython interface
The reasoning behind using AsyncIO is to show how easy it is to write a simple RPC based system and we are using ZMQ as the queueing system. The asyncio server is capable of handling my requests from client and it is preferable to not have lua code running in the same process as the server module. Also, it was also one of the things that I wanted to test, to see; if it is possible to have asyncio communicate via ZMQ and call Cython code. Since Cython is using threads a little differently than asyncio module, it made sense to keep them separate. Why ZMQ? The answer will be clear very shortly. Following is the code for the both client and server running from the same event loop. We could have created them separately in separate files, but it would have been inconsequential to the outcome of this exercise.

 import asyncio
 import zmq
 import aiozmq
 import uvloop
 import time
 import os
 import aiozmq.rpc

from lua_runner_cython import callCfunc

class ServerHandler(aiozmq.rpc.AttrHandler):

   def remote_func(self, a: int, b: int) -> int:
     return callCfunc()

  async def go():
     server = await aiozmq.rpc.serve_rpc(
     ServerHandler(), bind='tcp://*:*')
     server_addr = list(server.transport.bindings())[0]
     client = await aiozmq.rpc.connect_rpc(connect=server_addr)
     count = 0
     while True:
       ret = await, 2)
       if ret < 5:
         count += 1
         if count > 5:
     print("Returned from lua -> cythond -> asyncio:", ret)

   await server.wait_closed()
   await client.wait_closed()

def main():

if __name__ == '__main__':

As you can see, there is a single go() method, which is creating both client and server ZMQ sockets and then calling the remote function declared by the ServerHandler class, which extends AttrHandler from aiozmp library and is responsible for making a call to Cython code. The loop in which the call is being made to the remote function breaks as soon as the random number between the range 0 to 10 have upto 5 return values that are greater than 5. Once the loop terminates, we close both client as well as server socket and then we are done. On running this code you see the following output.

Returned from lua -> cythond -> asyncio: 9
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 9
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 9
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 9
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 9
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 2
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 2
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 2
Priming Run
address: (null), port: 0
Caling Random
Returned from lua -> cythond -> asyncio: 2
Priming Run
address: (null), port: 0
Caling Random

But there is one problem that we have not yet handled. It is the fact that the lua code is being loaded every time the call is made to the Cython function. This is very bad for performance as well as it does not reuse the pointers that are already created for each run. You can also see that the priming run is being repeated, which means, the lua code file is being prepped on every function call. This needs to go and go it will in the follow up article to this one. For now, we will halt with the ability to have an asyncio based call calling cython, which inturn calls the lua code via luajit.
I hope the readers enjoyed this exercise. Either ways, please leave a comment with any suggestions, improvements and corrections that there may be.
Until next time.

Asynchronous execution and celery musings

Lately, I had a task of creating a workflow manager and after much testing and feature evaluation of products such as Luigi, TaskFlow, Azkaban, etc. I settled on doing it with plain celery. I chose to use canvas functionality offered by celery and was floored by what I found. It was easy to create workflows using chain and group feature of celery and it was also very performant system. The configuration for system that was finalized had rabbitmq as broker and mongodb as result backend. Since the use case was to have completely asynchronous flows, there was hardly any need for result backend except for celery’s own use when handing subtask level errors. Since what we were building itself was going to partly do what celery does, the result backend using Mongodb seemed like a good choice. It enabled us to perform queries and aggregate results, with its powerful aggregation apis. However, there were two problems which needed addressing. Although, they are not a blocker but they really made me think of why celery wouldn’t be a very good choice in certain backend scenarios. It’s greatest strength of being very simple to use was also its greatest weakness, since it allowed us to use it badly. I didn’t get much time to spend in the internals of the source code of celery but it was a bit difficult to extend as well as to tweak it to whatever specification one may wish when implementing backend asynchronous systems.

To understand that let’s look at the example provided by celery to see what is it that I’m getting at. Following is the simplest getting started code from the celery’s quickstart page.

from celery import Celery
app = Celery(‘tasks’, broker=‘pyamqp://guest@localhost//’)

def add(x, y):
    return x + y

F = add.delay(10, 20) # Returns asynchronous future object to be used later
print(F.get()) # Block for the result, requires result backend to be configured

As can be seen from above, getting celery to make your tasks asynchronous is very easy. You can start a worker and call the last two lines from your code base, which will run add method on the worker. This requires that your code base must be shared between the calling code location and the location from where worker is run. It can simply be accomplished by running the same code base in separate locations as either master (to publish the task) and worker (to consume and execute those tasks). However, what about the situation where the master may not be part of the same code base. What if the call to add.delay() is being made from entirely different system, which does not have the implementation of add to use as celery task. This can be overcome by defining the method with the master, as shown below, and have the real implementation shown above with worker alone.

def add(x, y):

This works for a small project but what if the calls are supposed to behave like RPC calls and stubs need not be present at every location of master or the worker. In that case, the management of the tasks becomes a bit tricky and we must resort to git submodules to make everything available to everyone.

Another problem which was encountered is the fact that all tasks that needs to run as celery task must be decorated as shown in the example. We cannot make any method a celery task arbitrarily. Nor is it easy to use Object Orientation to create tasks without having to extend the celery’s Task object. If using celery is finalized and there is not turning back ever then it seems fine. But what about the cases where you may have to switch to some other task executor in the system (concurrent.futures?). It will cause the code base to change, since the separation of concerns and decoration are all done in a fixed manner on the code level. What I would really like is to have POPO (plain old python objects) define the tasks with some fixed method (duck typing) for making it a runnable task and send it to the executor without having to worry about which class it extends and whether it is decorated in a certain way. It does not stop there. If I must use celery then there must be broker to go with the system. What if I don’t need a broker? What if I need to execute the code in the same process asynchronously? How to leverage the fact that python 3.5 onwards we have a very great feature called asyncio? Can I choose or mix between local async execution versus remote execution using multiprocessing queue or third party messaging queue? How about zeromq instead of rabbitMQ? What about the design patterns involved? Should I be able to create a command and send it to executors which will raise certain events to be handled by creating another set of commands ran by the same executor, regardless of whether the executor is local or remote?

Like I mentioned earlier, I’m still learning celery so there is a possibility that most of the questions I have raised in previous paragraphs are already taken care of. But I would like to think of a different way, which would not depend on a library/framework that locks me to it. Which is why, I believe that the solution must leverage existing stand libraries like asyncio, multiprocessing, collections, etc and it must use design patterns such as command and observer besides, abstract factory and builder to accomplish the goal. This will allow the solution to be extensible and yet abstract enough to not change. Although, I do not have the answer to it yet, I have started a open source project to accomplish just that, which can be accessed here. Following is the class diagram for the way the system is being envisioned. I will follow up with the development and update on the blog as it proceeds.

As the system is envisioned it must have inline executor to execute the commands in the same process or future executor that can further be of two types; multiprocessing based or external queue based (preferably zmq since it does not involve a need for a broker). The command executor will raise events with listener being based on coroutine or on a separate process listening on a queue, either of which will initiate the event handler by creating the command and sending it back to the command executor. The command registry is where all the tasks will be registered with a key and executors will look up the executable part of the command from the registry. Since this registry will contain commands, which must be callable with “(*args, **kwargs)” arguments, in it to be registered. This way, using the duck typing afforted by python, we can add any object to the registry, giving us the freedom to register the command callables from anywhere in the system.

Of course, there is a room for improvement in this design, but it will also be capable for running flows where the commands can be composite and to be run in parallel or sequentially, with each of those commands being a simple command or a composite one. The behaviour of such system solely depends on the standard modules without the need for any third party apis and with the ability to use any callable in the system as a command, it opens up the possibility to not get locked in to any particular framework during the actual implementation.

Celery is a cool piece a software and I have already fallen in love with it. Therefore, it makes sense that I should be able to leverage that too, if I need to use it. As can be seen, the proposed design is extensible and celery itself can be added as a remote executor and be used with this system. However, the catch is that it reduces the celery to being a simple executor and gives up a lot of it’s great features such as canvas and signals, etc. by making them redundant. As I mentioned earlier, this is a proposed design, implementation is underway and if it makes sense then it is prone to change. Perhaps, I will end up making a celery clone after realizing that it was the best thing for this sort of work and that will kind of be like going nowhere, as far as this project is concerned. But then, I will have learned something deeper about celery and I don’t think that would be bad result.

Until next time.

The missing gap in AI revolution

As much as we have come far in the field of AI, there is this sure gap between how generic humans capabilities are and how focused the AI is, regardless of how better they are at those specific areas of expertise. Following is my perception of what could be missing and how would that gap be filled to achieve Artificial Generic Intelligence. All the tools of AI: be it probabilistic graphical models, regressions or symbolic solvers, have one thing in common and that thing is bad input. To understand this let’s take an analogy, what if I write it article and then zip it before sharing? Everyone who would download it will know to use decompression before viewing the actual text, which is a common knowledge. Nobody would read the zip file as is, by opening it using text editor and trying to make sense of what is shown. What this analogy does is explain why computers are having hard time understanding human world and it environment. The input we give to computers are texts, images, other medias, so on and so forth. What they really lack is the context of that information. All that data we feed in is compressed. And I do not mean JPEG compression, but rather the language itself is compressed. We as humans have done a great job of abstracting the concepts in words and when we transmit the text through words, we tend to compress the abstract concepts in those words that the recipient would use imagination to decompress and then understand it. Even our brain remembers understanding it not the exact details. What is happening in the field of AI is like training a child to learn English language using Morse codes.

What can be suggested as a better approach is to use real world environment to train the AI machines to understand the imagery, audio and textual inputs. Instead of using 2d images, it would be better to give 3d ones to help computers understand the the concept of near and far through the depth which is present in 3d images. Even better mode of giving input is the use 3d movies with subtitles to train the neutral networks to understand objects in 3d images and subtitles to associate words with audio and imagery for developing concepts. Much like a real child would learn. It can be simple concepts, but once the machine starts abstracting objects it views as instance with history and belonging to a particular class of objects through the existing tools of object recognition, it can enrich the machine’s capability to interact with it’s environment. None of these would require any new algorithms or techniques, in fact, the existing tools are more than enough. It is the encoding of input signals that need to be made more understandable by the machines. Add generic programming to the mix and we can have self evolving machines that that learn and adapt to its environment.

Although, this is a small write up for me to clarify my own thought, I intend to explore this further and get responses to further refine this approach. All inputs are welcome. And since this is my first time blogging or writing in a public space, I seek forgiveness for my ineptitude in my ability to express appropriately and concisely.