Jekyll: YAML lists vs Dictionaries

YAML is a simple human readable data format that is widely used across projects. It has implementation in various programming languages. I have been using YAML data with Jekyll where it’s a standard way of representing data. Jekyll uses data files which are used to display content in web pages using the liquid templating engine.

An interesting thing that I realised while reviewing a Pull Request in PyConf Hyderabad website repo. I got to know the precise difference between YAML lists and dictionaries and how it is easy to mix up between both of them in Jekyll.

YAML lists are lines beginning at the same indentation level preceded with a - (dash) with a space to distinguish each element. Lists are indexed collection of objects commonly known as the list data structure.

Jekyll Dictionaries or hashes or mappings are lines with a key value pair.

- Apple
- Banana
- Guava
Apple: red
Banana: yellow
Guava: green

In Jekyll we can use a for loop to iterate over elements of a list or dictionary. So if we save this list in a file _data/fruits.yml ( which is a standard way how data files are arranged in Jekyll ) we can use a for loop to print each item in html as show below.

{% for fruit in site.data.fruits %}
  <p>{{ fruit }}</p>
{% endfor %}

Here is the output of each of the examples

Apple
Banana
Guava
Applered
Bananayellow
Guavagreen

You can observe how Jekyll Concatenates the key value pairs for dictionaries. If instead we want to display just the key or value we can use the array notation where key = fruit[0] and value = fruit[1] Jekyll stores each key, value pair in an array of two elements. There is also another way using the dot notation which is more commonly used to represent data where the keys are decided beforehand and we use the keys to access the values.

- name: Apple
  color: red
- name: Banana
  color: yellow
- name: Guava
  color: green 
- {name: Apple, color: red}
- {name: Banana, color: yellow}
- {name: Guava, color: green}

Both of the formats above represent a list of dictionaries, the second one is is called the short hand dictionary notation which is very common to the python dictionary notation. The first one is the long format which is more commonly used to represent complex data and is more readable. We use the code below to iterate over the elements.

{% for fruit in site.data.fruits %}
  <p>{{ fruit.name }} is {{ fruit.color }} in red in color</p>
{% endfor %}

Ambiguity of Lists and Dictionaries

The confusion arises when we tend to use data with unique keys like the following example

007:
  name: James Bond
  work: Secret Agent
221B:
  name: Shelock Holmes
  work: Detective
- 007:
    name: James Bond
    work: Secret Agent
- 221B:
    name: Sherlock Holmes
    work: Detective

Here we represent some persons with some unique codes. We save it in file _data/persons.yml and access them with the following code

{% for person in site.data.persons %}
    <p><span {% if person[0] == "OO7" %} style="color: red" {% endif%}>{{ person[1].name }}</span> is a {{ person[1].work }}</p>
{% endfor %}

For the first case the code works fine and produces the following result

James Bond is a Secret Agent
Sherlock Holmes is a Detective

But for the second case we get an undesirable output. What’s the reason ? If you carefully observe there are dashes in the beginning of the line which implies it’s a list of dictionaries. So iterating over each element actually gives you a hash of the person object. The correct code to get the desired result would be

{% for person_hash in site.data.persons %}
  {% for person in person_hash %}
      <p><span {% if person[0] == "OO7" %} style="color: red" {% endif%}>{{ person[1].name }}</span> is a {{ person[1].work }}</p>
  {% endfor %}
{% endfor %}

This can be more clearly understood by revisiting the first example.

- Apple
- Banana
- Guava
Apple
Banana
Guava

If the below code is used to run both of the data then the result produced will be as follows

{% for fruit in site.data.fruits %}
  {{ fruit }}
{% endfor %}
Apple
Banana
Guava
Apple Banana Guava

The reason being in first case Jekyll interprets it as a list because of the dash in each line while in the second case Jekyll treats it as a dictionary as there are no dashes in each line. Since there are no key value pairs so it concatenates all the lines as a single key which is shown as the output.

The summary of the discussion is – whenever a line starts with a dash in the YAML data it represents a list of objects, if there’s no dash then it’s a hash with the specified key value pairs. There can be a list of dictionaries or a dictionary with a list of values. So you need to be careful while dealing with these type of data.

Serving multiple websites from sub routes of a domain using Nginx

Few days earlier I was working on migrating the PyConf Hyderabad website from GitHub Pages to a DigitalOcean server in order to achieve archive support for the previous year and current website. The websites are static websites generated by Jekyll.

Objective

The objective was to serve various years of the conference websites in year wise sub routes. The domain was pyconf.hydpy.org . So we wanted something like pyconf.hydpy.org/2017/ , pyconf.hydpy.org/2019/ . We had two separate repositories for both of the websites which we wanted to serve in the stated manner.

Solution

Nginx directives have an excellent way of accomplishing this by leveraging the power of regular expressions. Nginx is being widely used for most of the production level web servers. I was also using Nginx as the server in the DigitalOcean droplet.

Since the websites were in two different repos all I had to do is – whenever a request comes with a specific subroute I would point it to the specific directory path. This was done in the Nginx location directive. I had to change the Nginx config with the following similar code

location = / {
    rewrite "^.*$" /2019/ redirect;
}

location ~ ^/2017(.*)$ {
    alias /home/pyconf/hydpyconf2017/_site/$1;
}

location ~ ^/2019(.*)$ {
    alias /home/pyconf/hydpyconf2019/_site/$1;
}

Let us understand this step wise. There are two parts of this configuration

Step 1

Redirect home url to /2019/ domain . This is to redirect the root domain to the sub route of the present year website. The = modifier after the location directive matches the exact route which uses the rewrite directive to redirect it to the specified replacement expression.

Step 2

The ~ modifier is used for case sensitive regular expression matching. Here we specify that if we match a route starting with /2017 search that path in the directory specified by the alias directive. The $1 refers to the first matched group in the regex. We can use the groups in regex to form the replacement strings that we want to point to.

Thus you can see with the help of regular expressions we are able to serve two different static websites using Nginx directives.

Science Hack Day India 2019 Experience

© Sayan Chowdhury

This was my first year attending Science Hack Day India. I came to know about Science Hack Day after I met Praveen at FOSSASIA Summit 2017 at Singapore. He started the Science Hack Day (aka SHD) event in Belgaum, the city where he lives. This year was the 4th Edition of SHD India. The idea of Science Hack Day is to make anything in 24 hrs using Science. But it’s actually more than than. You will get to know how it turns out at the end of this post.

Day 1

I booked my tickets for SHD India 2 weeks before the conference. I reached the venue on Saturday, 19th Oct. My bus was 2 hrs late. The venue was in Sankalp Bhumi Resort in Parvati Nagar, Belgaum where SHD India is held every year. Once you reach the team picks you up from the bus stop to their venue, but since I was late Praveen guided the Auto driver to take me stratight to the venue. After I reached I met few students who come each year to the event. I met Sayan at the Breakfast area which they call ‘Dhaba’ . After Breakfast I went to my room to freshen up and get ready for the event. The venue of SHD India is really picturesque. It’s calm and quite place full of greenery around with some awesome spots for Hacking ! Here is a image I took of the cottage house I was living.

I went to the stage where the event was about to start. I met Jithin and Saptak there. They were going to work on a robotic arm powered by KuttyPy. Jithin is the maintainer of KuttyPy and he showed how it is used to send signals to a output port of a ATmega32 microcontroller by simple clicks of a button using the KuttyPy gui.

The event kicked off by introducing Science Hack Day. After that Mentors introduced their hacks to the audience and welcomed everyone interested to join their hacks. I introduced about the Papercraft Spectrometer by PublicLab which I was going to make for the event. There were many young hackers among us who were excited to Hack among us and also introduced their Hacks. After that I went to the Hacking area at the ‘Cafeteria’ . I met Hitesh who was hosting a CTF . It was the first time a CTF event was being organised at SHD. Me, Sayan and Saptak started solving the CTF puzzles. The problems were intriguing and we kept solving them one by one. Other hackers were working in some interesting stuff. I met Akshai M who was working on a LoRaWAN Transmitter. We breaked for lunch around 2 pm. The weather was cloudy and it was raining for most part of the day. Post lunch there was a Python 101 Workshop for college students hosted by Sayan. I helped people around to Install Python in their systems and writing their first Python Program. After that I continued solving CTF questions while others continued their hacks. Here is a photo of everyone busy Hacking 🙂

In the evening there were Lightning Talks. There were all sorts of lightning talks starting from Hardware Systems, Security and Privacy over Web to Gardening plants. I gave a lightning talk on my HydPy community experiences on how regional communities play an important role on engaging more people towards a shared objective and idea. It was the end of the day. We had Dinner after that where we had more interesting discussions. I returned to my room since it was raining quite a lot, where everyone (Me, Praveen, Sayan, Saptak, Akshai, Hari, Jithin) gathered beside the balcony of the cottage house discussing their experiences in FOSS and Conferences. It was bed time soon, end of an exciting first day.

Day 2

The next morning we had a video conference with the SHD San Francisco Team at 9.30 am. SHD was also happening in the same days at San Fransico this year. I woke up and went for breakfast with Sayan. The item served for breakfast was unique for me – Mishar and bread, a typical Maharashtrian breakfast item. After having breakfast everyone gathered near the Cafeteria for the video call. Ariel who is the global Director of SHD was hosting the call from San Francisco. Everyone was excited to share their hacks they were working on. It was really amazing to see people from 2 different timezones sharing there awesome hacks and their excellent level of enthusiasm. After the call ended everyone started working on their hacks again. They were to be presented soon to the audience so there was little time left. I went to take some materials for my Papercraft Spectrometer. There was a team which made the Da Vinci Bridge using wood. People were walking over it turn by turn. I also had a walk over the bridge. Hari was taking a Unicycle riding session for people.

Meanwhile there was a microcontroller Workshop using KuttyPy going on in parallel which was hosted by Jithin. My hack was almost complete and I was seeing hacks of other people. A team made an awesome Ruben’s Tube where we could see flames dancing with music. Shortly after everyone gathered for the SHD India group photo in the lawn. There was a Treasure Hunt event after that. Reminded me about my college days. Hackers were giving final touches to their hacks which were to be presented after lunch.

After lunch everyone gathered near the stage and get their hacks ready for the audience. I got Table 1 where I was talking to people Regarding Spectrometry and my Spectrometer hack. It was nice to talk to people from all age groups. There were some really good hacks around. Praveen was demonstrating Hacks using Liquid Nitrogen, breaking flowers after freezing them with Liquid Nitrogen. It was really spectacular to see science Lab experiments done in fun but careful way.

After the demonstrations were over it was time for Awards for the Best Hacks. The Da Vinci Bridge and Ruben’s Tube were among the Award winning Hacks. The Best Hack of the event was given to a Team who made colourful clay plasters . There were more Awards coming in. Me, Sayan and Saptak were jointly awarded the CTF Award for scoring 950 points individually. I got a medal for that 🙂 Finally it was time for the closing address, thanking all organisers and Sponsors and See everyone at next SHD. Fun events continued after that. There was a Domino sequence made by some students. It was really nice. Then there was Bubble blowing by participants. Enthusiasm was still high among the participants and Hackers.

Also it was time for me to say goodbye to friends at SHD. I had my bus at 9.15 pm. It was raining quite a lot. Before I left for my room for packing my things I spoke to Akshay Deshpande who was a mountaineer and doing Treks at Himalayan ranges. He was sharing anecdotes about his mountaineering experiences. He took a session on Mountain Raffling in SHD. Then there was music and lights and people dancing together. I had my dinner and after saying goodbye to everyone at SHD it was the end of event for me. Praveen arranged by drop at the Bus Stop. He has been a Super Host !

I really loved my first Science Hack Day. It was a weekend full of Hacks, Fun and exciting discussions beside the tranquil venue at Sankalp Bhoomi. Making friends is the best part of the event which I look out whenever I go to Conferences and meetups. Eagerly waiting for the SHD India event !

PyCon India 2019 Experience

What an experience it was! This was my second time attending PyCon India and third time attending a Python Conference. I have been a part of the organising team of PyConf Hyderabad 2017 and PyCon India 2018 (Hyderabad). Each time it has been a fulfilling experience and numerous takeaways and new bonds of friendship made. I haven’t blogged about my experiences in past conferences. So this will be my first blog post regarding my Conference experience.

The last 2 times I had attended Conferences close to where I was presently living. This time PyCon India was held in its 11th Edition at Chennai. Being organiser in past Conferences I knew about the wonderful experiences that I was involved, so I decided to be part of the Volunteering team for this year’s PyCon. I was eagerly waiting for the event from when it was announced. PyCon had become a mandatory event for me now, where I get to meet old friends and make new ones.

Day 0

I reached Chennai on 11th Oct night, before the Conference to help out with volunteer activities. I went straight to the Conference venue at Chennai Trade Centre where many volunteers had gathered for helping out with Pre Conference activities and goodie bag packing. This is like a fun event where we introduce ourselves and assign tasks for the upcoming days. I met HydPy folks – Ram, Murthy, Gokul; Dgplug friends – Rayan, Devesh, Chandan, Bhavin and many more friends whom I met at last PyCon. Chandan was co-ordinating the Volunteer team. I also met Vijay, Naren and folks from Chennaipy who were the organisers this year. After Dinner we started packing the goodies bags and sorting Attendee ID cards and getting prepared for the big day. It was almost 2 am when we left the venue to our hotel. Unfortunately my Oyo booking was cancelled, thanks to Murthy who shared his accommodation for the night.

Day 1

It was the first day of the Conference. I reached the venue around 8 am, there were already people coming in for registration. So I sat at the Registration Desk to help with Registration. I met with Kuntal (@hellozee) from Dgplug and Rayan who helped with the Registrations. Within no time there was a big queue of people. After sometime when the crowd lessened I went for Breakfast where I met Chirag, who came for the day. We discussed regarding the Upcoming PyConf Hyderabad 2019 event. After that I met all folks from HydPy who had come up for the conference. Here’s our group photo

After that I went to attend Pradyun‘s talk on Python Packaging . It was a really insightful talk explaining the pip ecosystem. Soon after the talk I started going around Sponsor booths to know more about their business and also to solve the puzzles they were gibing out to get some goodies 🙂 This year PyCon India had a long list of Sponsors – 41 of them ! Kudos to PyCon India team!

Post lunch there was Poster Presentation Scheduled where I was going to talk about HydPy Community. Also as I was a volunteer helped in setting up for Poster sessions as well. Folks from HydPy were also there, we also talked about our upcoming conference – PyConf Hyderabad on Dec 7-8. The day got over with ASL Devi‘s Keynote regarding Bridging gender gap in tech. It was an eye opening talk on different gender stereotypes and the unconscious biases that we have in our mind and how we can overcome this together. After that we had Volunteer’s meet to retrospect and discuss feedback regarding the day. The day ended with the Volunteer and Speakers dinner.

Day 2

My Day started again in the registration desk where I sat down to help with Lightning Talk registration and giving out ID cards to the attendees who missed taking it last day. Shortly after I went to the First Keynote of the day by Ines Montani . The Talk was titled Let Them Write Code . The Talk covered some best practices in development and coding, her Lessons from Open Source and much more. We had the Dgplug Staircase Meeting today which is a customary meeting that we have in PyCon India each other where we meet and greet people of Dgplug Community whom we have always seen in #dgplug IRC channel. It’s the time when we actually meet and talk to these people we see online. Kushal‘s absence was felt this time. Sayan was hosting this meeting and he discussed some important concerns regarding the community and the need for people interacting more in the IRC channel. After that he distributed Dgplug T-shirts which we had ordered before the Conference. We had the mandatory Dgplug group photo after that 🙂

I was helping out at the Helpdesk mostly rest of the day and interacting with people whom I met. There was the PyLadies booth just beside the Helpdesk where I talked to women from PyLadies Chennai community. They had a PyLadies Lunch last day and a Speed Mentoring Session to invite more Women in Tech. Finally we were up to the Final Keynote of the day by David Beazley . The Talk was titled – A Talk Near the Future of Python. After he started his Live Coding session in the Keynote the audience was simply going ga ga over his performance. He Live coded a Stack Machine which he later turnd into a Web Assembly compiler which played a Rocket Game at the end. He showed Python has a bright future by demonstrating the power of Python + Webassembly. This was The best Talk I have ever attended till date. He already published the screencast. Do have a look and amaze yourself. The thing that most impressed me most was his fearless stunt in front of 1200+ attendees and calmly handling the intermittent crashes and debugging them.

Finally PyCon India ended with Vijay‘s closing Address who was the Chair for this year’s PyCon. It was the end of the Conference after which we there was Workshop + Devsprints for next 2 days. At the end we had group photo with all the attendees.

PyCon India has been successful for the large pool of people who volunteer at the event and come from all parts of the country for the love for the Community. It has been a pleasure to work and interact with these amazing people. I met Noah this year who came all the way from Taiwan and volunteered from Day 0 for the event. It really amazes me to see the enthusiasm of the people for the Community.

Day 3

Oct 14-15 were Workshop + Devsprint Days. The venue was IIT Madras Research Park. I was also leaving tonight so I packed my bags and went to the venue in the morning. I started helping with the registrations for Workshops and Devsprints and guiding people to the respective rooms. Then I went to the Devsprint room to work on the Python Packaging Sprint which was hosted by Pradyun. I was eagerly waiting for that. I started with the setup and then working on a issue. After Lunch I found David Beazley in the venue so I quickly joined for a quick conversation with him. I asked the secret behind his Live coding stunt last day. He said he did roughly 15 iterations of the run, even spent 45 mins to debug an issue in one iteration. But yeah the fluency came over his 35 years of experience ! Also as he frequently took classes with group of students where he used to live code during his classes. It was a real pleasure to talk to him. After Lunch I continued to work on another issue from pip till the end of the day.

Finally it was time to say goodbye to friends and the awesome 3 days taking back lots of memories and experiences. PyCon India has become an event close to my heart now and it’s an event I don’t want to miss. See everyone again in next year 🙂

Summary

PyCon India has let me meet and communicate with people from all parts of world and domain. There is something to learn in each of these events. It’s An event by the Community. For the Community. My takeway this year was – Work, Interact and Share your experiences. You can always be a good at coding and be an excellent programmer, but unless you interact with more people and share your learning you cannot be a good person. Also Volunteer and give back to the Community. Volunteering requires time and effort, but its a gesture to give back to the Community and meeting people who are doing amazing work for the Community. The entire Open Source world largely revolves around people volunteering their time for the good of the Community. So even a small effort towards that can make a big change.

That’s all about my PyCon India 2019 experience . Do leave your comments ! And if this excites you do come next year to PyCon India 🙂

First BangPypers Meetup

BangPypers , the Bangalore Python User Group is one of the largest and oldest Python User Group in India. It has been running successfully for more than 10+ years. Most of the known faces in Indian Python Community have been a part of this meetup group at sometime or other.

After I moved to Bangalore I have been on the look out to attend the meetups but I missed it in the first 2 months since some work used to come by on the meetup days. Last Saturday, 21st September, I finally made time to attend my first BangPypers meetup.

The meetup was schedule at 10.30 am at Visa Technology Centre, Bagmane Tech Park. I decided to go by Cycling since I was also looking for a long ride in my Cycle for a while. But alas it took longer than expected owing to a wrong turn in my way.

The theme of this week was Design Thinking and it had 3 Talks scheduled. I reached around 11.15 am almost missing the first Talk which was a Design Patterns 101 Talk. The Speaker was almost in his closing notes. Nevertheless I found a place to seat in the almost full room.

I took out my phone to take notes and waited for the next talk which was on Design Pattern for distributed Architecture . The Talk covered various points on best practices related to distributed systems and tools and techniques to achieve them. Discussion revolved around Horizontal Scalability, Responsiveness, Security, Failure Handling, Centralised Logging, Metrics, Request Tracing, Health Checks, Configuration and Discovery. Since I never dealt with production level Python applications or Distributed systems the terms were quite new to me, few of them I encountered in past but didn’t quite fiddle in depth. I made a note of the discussion for future reference.

The last talk of the day was Organize your bookshelf using Micropython by Vinay Keerthi who was one of the Hosts at Visa, which is also a PyCon India 2019 accepted Talk. He made a LED book shelf organiser which would tell him the position of a book in the shelf by lighting LED array. He used Micropython on a NodeMcu in which he built a simple Flask app which would fetch data from a Postgresql database and store it in a queue which would send signals to the LED. I really liked the idea and it was a really hacky way to find a book. After his presentation I talked to him to get some pointers on starting my first Micropython project which I plan to do in the upcoming days.

After the session I talked to the organisers Anirudha and Abhiram with whom I shared my experiences at Hyderabad Python Community and also how they conduct BangPypers meetups. Visa had arranged lunch for the attendees so the discussion continued at the Lunch table where I got to talk with other attendees and interacted about the work they were doing. Bangalore has a vibrant developer community and there are many more communities like PyData, PyLadies which conduct regular meetups. PyLadies Bangalore are going to have a meetup on coming Saturday, September 28. Here is the announcment about the same.

At the end Vinay shared some more stories of his Hacks and experiences with tinkering with Micropython. And yeah here are some photos of the meetup I took

The thing that made the meetup nice was the great Venue and the Topics of the Talks. With the boom of ML/AI in Python Land having Talks that are unlike those, but most fundamental topics is good. Hope to attend future meetups as well!

Lastly I rode back home completing my first 20km streak in Cycle 😛

I also tweeted about the meetup after I returned home that day.

Takeways from the Meetup

  • Design Patterns in Python – Spend more time learning these.
  • Got to know about best practices in distributed system – Keep notes for futute projects.
  • Micropython getting started – Order some NodeMcu and get the hands dirty. TO DO: Read blogs regarding starter projects.

DevConf India 2019 experience

One month ago I attended DevConf India 2019. It was held from August 2-3 at Christ University, Bangalore. It was quite a while since I attended a conference, the last one being PyCon India 2018 . Due to my laziness in writing I have altogether missed writing my Conference experiences till now. This will be in fact my first Conference blog post. Now on I will make sure to write about each of the conferences I go.

I came to know about DevConf India last year when it was first held, from people in my developer circle. DevConf India is organised by Red Hat and has similar events in US and Czech Republic. Since I was in Bangalore this year I made sure I attend it after I saw the dates in Twitter. I registered as soon as I came to know about it.

Day 1

I mostly planned on meeting up with people and attend a few talks. I started at 8.30 am from my place and unfortunately missed the Keynote owing to a bad experience with a Bike Taxi service. I reached the Venue around 10 am and collected my Attendee badge and T-shirt. Then I headed towards the Keynote Session Hall where I met Naren from Chennaipy. I earlier met him at PyCon India 2018. It was nice catching up with him.

After having breakfast at the venue I headed to the Booth area where I met Chandan . I started visiting the booths asking questions about various projects like Fedora, Debian, CentOS. Shortly after I met up with some more familiar faces from DgplugSayan, Rayan , all of them I met during PyCon India last year. I expected a Dgplug staircase meeting at DevConf but unlike last year there were less attendees this time. After that we went for lunch at the Cafeteria where I met Ramki , Praveen and pjp . Few days earlier I was reading pjp’s tutorial regarding gcc and gdb from dpglug irc logs . It was nice to catch up with him in person. I was discussing with Sayan and Praveen about the initial days of dgplug at my college at NIT Durgapur, attending their first talk in 2013 when I had just joined my college.

After Lunch I decided to attend few talks. I attended a talk regarding Evolution of Containers – there I came across terms like Chroot, Cgroups, Namespaces , how the whole container ecosystem was born. I have been always been inquisitive about containers and though I haven’t really worked on containers before this talk really fascinated me to dive into the world of containers.

Then I attended a talk on What hurts people in Open Source Community . The talk helped to set my expectations right regarding contribution to a Open source project and Community.

After that I went to the Closing Keynote of the day shortly after which we went for evening snacks where we had more discussions over Coffee and Dosa – we noticed a item mentioning ‘Open Dosa’ over which we laughed a lot 😛 . And it was finally close of the day.

Day 2

I reached a little late to the venue and went straight to the talk that I didn’t want to miss. It was a Documentation BoF where speakers were discussing how to create effective documentation and tools for creating collaborative documentation. I came across User Story based documentation and tools of the trade like asciidoc and asciidoctor . I met Ladar Levison there during that session and talked with him regarding better project organisation. He gave me his business Card which mentioned Lavabit . Little did I know about him until I read this article which explained more about Lavabit and his role in Snowden’s secure email communication. But that was after this conference and I wished I could talk more about Privacy and Lavabit projects.

After that I went for lunch with Sayan, Chandan and Rayan where we chatted on lot of different stuff on open source, food and conferences. After Lunch I went to attend Sinny‘s talk on Fedora CoreOS whom Sayan introduced last day.

Finally it was nearing the end of the day. I went to attend the closing keynote by Jered Floyd and sat beside Christian Heimes from Red Hat who was sharing anecdotes from his travel experiences.

Notes from the Conference

I made few notes that I would like to share from my experience at the Conference and also as a note to me for future Conferences

  • Try to search about people you meet so that you can know more about them. You may not know everything about the person you are talking with. But actually the person can be a mine of knowledge. Ask for the person’s email/Twitter so that you can follow up on email or Twitter after the conference.
  • It’s always good to prepare some questions if you are likely to meet a person you met/knew online. You have the opportunity to talk face to face and ask about the projects the person works on. You can even do that within the conference when you are free.
  • When you attend a Talk ask good questions that can start a conversation. Usually people take a interest in following up after the Talk with you and you get to talk to more people.
  • It’s always good to be Speaker at the Conference. That way there is higher chances that you can start a conversation with people you don’t know and meeting for the first time. This is something that I really need to work on and hopefully I will be able to submit a talk in the next Conference I attend.
  • When going for Lunch tag along with a group so that you get meet more people. If you are an introvert this really works well as you can meet friends of friends and you can interact much easily!

And yes don’t forget to take pictures 🙂 It really bring memories. It may sound weird but this is something I really forget every time I meet up with people and later wait for Conference photos.

A New City, A New Beginning

Two and a half months earlier I shifted to Bangalore after a 2 years stint at my first Company at Hyderabad. I was looking for new opportunities and started appearing in interviews when I hit the hard realisation that what I learned till now was not enough. I need to self-learn more and interact more with people and learn from stories that people experienced. I landed up a job at a EDA based firm at Bangalore and decided to move in there.

Leaving the city, but not the memories !

Time really flies and it had been already 2 years working in the Software Industry. I met with a lot of people, made friends and shared some good experiences. I became aware of the meetup culture, realised writing code is not the only way that makes you a good developer, realised that communication is a key factor in conveying your ideas.

I became part of the HydPy Community and organised 2 conferences – PyConf Hyderabad 2017 and PyCon India 2018 . I met some like minded people passionate about technology and flourishing the Python Community at Hyderabad. I still am a part of the Community and hope to continue doing so.

I also attended the dgplug Summer Training in 2018, made many friends there as well whom I keep meeting during conferences. They are a really amazing community and there is always something to learn from each of the IRC conversations.

I interacted with a lot of people from Python and open source Community. Going to Conferences and meetup is really a good way to interact with some awesome people and sharing their knowledge. But it’s also true that you really need to work on something tangible and develop your skills. Then only you can move forward and also put the experiences to use.

I have been living in the heart of the fast growing city at Hyderabad. It’s been a nice experience here as a whole (except from traffic on rainy days!). I explored to places around in the initial days. Some significant events happened during my stay – the GES 2018 was held, Hyderabad metro rail was inaugurated, the first IKEA store in India opened up . I Spent my first Durga puja away from home. Realised that Kolkata Biryani is still better than Hyderabadi Biryani. Saw more buildings, Tech parks and flyovers being constructed in Rapid phase. Got to experience the hot summer and pleasant winters. All together it had been a good time in the city of Nizams. Hope to come again here someday !

The New Phase, What Next ?

It’s been 2.5 months while I came here at Bangalore. It’s called the Garden City of India because of the amount of greenery it has. But I would rather call it a City of Traffic ! Dealing with traffic can be terrifying here. Nevertheless there is a lot of greenery left and I see palm trees here and there quite often. The area I live is booming with an array of multinational companies, I also observed more number of semiconductor based companies. Rightly calling it “Silicon Valley of India”. It’s pretty early to state my experience in this city so maybe I will write about it in the next phase 🙂

So what next ? I want to make my time here more productive and also develop some good habits that I have been lingering about. Get into the habit of writing, take time out for more self-learning, contribute to open source more often, interact with more people, put the experiences to use as much as I can, get more exercise and cut laziness. In fact I got me a cycle and have been doing my daily commute with it! I know now it’s more talk than work now. But I want this blog post as a reminder so that whenever I wander off I can come to this page and find what I need to do! And keep myself prepared for the next phase.

Adding Print Preview Support to CEF

Chromium Embedded Framework (CEF) is a framework for embedding Chromium-based browsers in other applications. Chromium itself isn’t a extensible library. It helps you to embed a chromium browser in a native desktop application. The project was started by Marshall Greenblatt in 2009 as an open source project and since then it has been used by a number of popular applications like Spotify, Amazon Music, Unreal Engine, Adobe Acrobat, Steam and many more (full list can be found in this wiki). CEF is supported for Windows, Linux and macOS platforms.

There are 2 versions of CEF – CEF1 and CEF3. CEF1 is a single process implementation based in Chrome Webkit API. It’s no longer supported or maintained. CEF3 is a multiprocess implementation based on Chromium Content API and has performance similar to Google Chrome.

Preface

The purpose of writing this article is to document the work that I did while working on the CEF project. The major focus is on the Print Preview addition to CEF upstream project where I worked on.

For the past 1 year I have been working on CEF as a part of my day job. Initially the work was to maintain updates of CEF with every Chromium version released. We had a fork of the CEF open source project in which we applied some extra patches as per the requirements of the custom desktop application that used it. Building CEF was quite similar to building Chromium. It had a build script which was mostly used from downloading the code, building and packaging the binaries. Everything is documented here in this link . Upgrading the CEF was quite some task since building CEF took a lot of time and resources (a lot of CPU cores and a lot of Memory) and since CEF was based out of chromium I had to skim through parts of chromium code. Chromium has a nice developer documentation and a good code search engine that eased a lot of things. But owing to the fact that chromium has a huge codebase the documentation was outdated in few areas.

Feature Description

The interesting part of the CEF project came when I was handed over a work of a missing piece in CEF. Chromium supports in browser print preview where you can preview pages before printing, similar to one shown in the picture below.

CEF didn’t support this feature and had the legacy print menu where you cannot preview the pages to be printed.

This meant applications that used CEF couldn’t support print preview within them. The task was to make print preview available in CEF.

Initial work

The work started with CEF version 3112 (supported chromium v60) and was in a working state in CEF 3239 (supported chromium v63) in our CEF fork. Then the change was supported only in Windows since our desktop application that used it was a Windows only application. I was handed over the work in CEF 3325 (supported chromium v65) where the following specs already existed in the Print Preview patch. The relevant blocks of code is available in the CEF source code now.

  • enable_service_discovery is disabled now
  • CefPrintViewManager::PrintPreviewNow() will handle print-preview
  • CefPrintViewManager will not handle the print now. It handles the PrintToPdf function exposed through browserHost. Also, it listens to two print preview message PrintHostMsg_RequestPrintPreview, PrintHostMsg_ShowScriptedPrintPreview and generate the browserInfo for print preview dialog
  • Define interfaces from web_modal::WebContentsModalDialogManagerDelegate and web_modal::WebContentsModalDialogHost required by the constrained window of print preview dialog
  • Define platform specific definitions of GetDialogPosition() and GetMaximumDialogSize() used by browser platform_delegate
  • Register Profile preferences required for printing
  • Add ConstrainerWindowsViewsClient class which is called in browser_main.cc
  • CefPrintViewManager::InitializePrintPreview() initializes the print preview dialog which further calls PrintPreviewHelper::Initialize() which generates the browser_info required by print preview
  • Remove CefPrintViewManagerBase and its associated methods from CefPrintViewManager. Those methods are redundant after the print preview changes
  • Check for switches::kDisablePrintPreview in CefPrintRenderFrameHelperDelegate::IsPrintPreviewEnabled() to determine whether print preview is enabled
  • Add chromium patch to fix errors in debug build

My Contribution to CEF Print Preview

After I took over the CEF print preview work I did a number of changes to print preview, specs of which is documented below

  • Add print_preview_resources.pak in BUILD.gn to fix blank screen error which came from chromium v72 onwards because of updated print preview UI
  • Add PrintPreviewEnabled() to extensions_util
  • Add switches::kDisablePrintPreview to kSwitchNames in CefContentBrowserClient::AppendExtraCommandLineSwitches() to disable print preview on using --disable-print-preview command line switch
  • Remove print_header_footer_1478_1565 chromium patch since it’s no longer required to disable print preview by default
  • Add WebContentsDialogHelper() to wrap web_modal::WebContentsModalDialogManagerDelegate and web_modal::WebContentsModalDialogHost interface methods. Move it in a separate header and cc file
  • Add support for disabling print preview via chromium preferences
  • Disable print preview for OSR mode
    • For ui print this is done by using CefBrowserHostImpl::IsWindowless() method
    • For scripted print this is done by passing the is_windowless parameter in CefPrintRenderFrameHelperDelegate object from CefContentRendererClient::MaybeCreateBrowser() method in content_renderer_client.cc
  • Fix DownloadPrefs::FromBrowserContext for CEF use case. Add GetDownloadPrefs() to CefBrowserContext and use that to get the download_prefs in chromium
  • Add extra_info param to CreatePopupBrowserInfo()
  • Fix MacOS build. Add GetNativeView() method for MacOS platform delegate and update GetMaximumDialogSize() method
  • Disable print preview for MacOS (in extensions::PrintPreviewEnabled()) since the print dialog crashes on print
  • Disable print preview if pdf extension is not enabled
  • Use CEF file chooser instead of chromium while saving to pdf in print preview. Add ShowCefSaveAsDialog() and SaveAsDialogDismissed() to PdfPrinterHandler.

Challenges faced

Integrating print preview was a big and non-trivial change in CEF since not only it needed good understanding of the printing code in chromium but also the print preview feature was getting constant updates from Chromium. The code was constantly changing with every chromium version released and the print preview chromium documentation was outdated.

CEF3 has a multiprocess architecture similar to chromium’s documented here . There is a main browser process and multiple renderer processes. Debugging multiprocess applications can be trickier. I used Visual Studio which made things a bit easier as it has the Child process Debugging power tool which is a extension that would automatically attach Child processes and helped to debug into the child processes whenever they spawned up.

The chromium v72 version introduced a new print preview UI which broke the renderer, we got a blank screen in print previw. It took weeks to figure out what was wrong. Finally it came out that a pak file was missing which needed to be included in BUILD.gn. I had to spend multiple debugging session with my team to figure that out.

Also it had to be supported for all platforms (Windows, Linux, macOS) to qualify to be merged to the CEF upstream repo. Each platform had a different way of rendering dialogs. Though the windows support was working the Linux and MacOS weren’t supported in the changes yet. I added the support for Linux platform after building CEF in a linux VM. The MacOS support finally didn’t work out and we had to keep using legacy print for Mac platform. Though I needed to ensure the change built fine in Mac, so I had to build it for Mac as well (I was given a separate Mac machine just because of this since Mac doesn’t ship VM images) and in fact the change broke the MacOS build so the issues had to be fixed.

Conclusion

Even after all these changes the functionality broke after a architectural change was made in CEF in version 3770 (supported chromium v75) in this commit which rendered blank screen during print preview. Marshall took over the work from there and made a number of changes in the patch which can be seen in this PR chromiumembedded/cef#126 . The change was added manually in master revision 1669c0a on 20th July. It will be supported from the next CEF version (supported chromium v77). The current implementation supports print preview in Windows and Linux platforms via a --enable-print-preview flag.

Overall it has been a good experience working in the project and I got to know a lot about chromium itself. This was my first major contribution in a C++ based project. It helped me to understand how a browser works under the hood, how a browser renders web pages and processes javascript. I hope to carry forward this knowledge in some similar browser based project in future.

Rescuing GRUB2 from rescue mode in Fedora 30

About 3 months back I installed the newly released Fedora 30 OS – dual boot with Windows 10 in my PC. This blog post comes from the notes I made during that time and as I troubleshooting note for future.

I had Fedora 29 and Windows 10 dual boot in my PC before that. The Fedora install partition was running out of space due to low disk space selected during my last install so I decided to do a clean reinstall this time. I made a live usb using the Fedora media writer for windows and the Fedora 30 iso available at getfedora download page. I followed the usual steps that I followed for installing earlier Linux installations in my PC, similar to what mentioned in this video.

The installation went fine and finally I was ready to boot from my hard drive. Then I saw what is called the Dreaded GRUB2 boot prompt.

error: unknown filesystem.
Entering rescue mode...
grub rescue>

First Attempt for Fix

I quickly started finding way to fix the grub. The first thing I found was the steps listed in this video. I had to choose the right partition from which the bootloader would load.

grub rescue> ls
(hd0) (hd0,msdos4) (hd0,msdos3) (hd0,msdos2) (hd0,msdos1)

This shows the various partitions in my hard drive. One of this is the linux partition where my Fedora 30 OS is installed. I need to list all the partitions and one of them will have the linux filesystem.

grub rescue> ls (hd0, msdos4)/
bin/  boot/  dev/  etc/  home/  lib/  lib64/  lost+found/  media/  mnt/  opt/  proc/  root/  run/  sbin/  srv/  sys/  tmp/  usr/  var/

Now I ran the following commands and waited for the system to boot up

grub rescue> set prefix=(hd0, msdos4)/boot/grub2
grub rescue> insmod normal
grub rescue> normal

I got the same grub rescue boot prompt but this time with a different error

error: file '/boot/grub2/i386-pc/normal.mod' not found.
Entering rescue mode…
grub rescue>

Second Attempt ..

The issue was that, the i386-pc folder was missing in the /boot/grub2 folder. The fix that I found for the issue was related to grub2 not being properly installed at the boot location. Luckily I was able to boot Fedora from UEFI Boot option from the Boot Menu. I logged into Fedora and reinstalled grub2.

$ sudo grub2-install /dev/sda
$ dnf install grub2-efi

I hoped that this would fix the issue, but it again came down to the same starting point loading the grub2 rescue prompt.

Third time is the charm !

I further searched and landed up in the fedora grub2 manual . After reading it I realized there is something wrong in my grub2 configuration. I booted into my OS using UEFI boot and opened /boot/grub2/grub.cfg file. The entry for Windows was missing. I followed the steps given in this section. I went to the grub rescue prompt and fired the following commands

grub rescue> set root=(hd0, msdos4)
grub rescue> linux /boot/vmlinuz-5.1.20-300.fc30.x86_64 root=/dev/sda4
grub rescue> initrd /boot/initramfs-5.1.20-300.fc30.x86_64.img
grub rescue> boot

Then I recreated the grub.cfg file using these commands

$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ grub2-install --boot-directory=/boot /dev/sda

Voila ! I was able to see the grub menu with all the boot entries.

Postmortem Report

So why did the issue actually occur ? It didn’t happend in the past whenever I did a fresh installation nor it’s a issue specific to Fedora 30. I tried to dig the actual cause of the issue and after a little finding and introspection I came to this conclusion.

Using the Fedora media writer was in fact the place where I unknowingly did a mistake. I usually used UNetbootin for creating my linux live usb in past which made the images to boot in BIOS only mode. The Fedora media writer enables to boot in both UEFI and BIOS mode. My Windows installation boots via Legacy boot and it has always been like that and since I have been using UNetbootin earlier, it always picked up Boot via BIOS for the live images. This time while creating the Fedora 30 image using Fedora Media writer the default boot mode picked up was UEFI and that created a EFI GRUB2 configuration. Now when I booted the live usb I just booted from the “Boot from USB” option without noticing whether it was UEFI or BIOS  and I went ahead and did Fedora 30 installation. Now my default boot option was Legacy Boot (since it supports the Windows boot) while the installed Fedora OS grub loader was created to boot in EFI mode. That in turn caused this problem due to a corrupted grub2 configuration.

Moral of this Story

Always be careful while created OS images. Check how it is supposed to boot. In case of dual boots all the OS must boot via the same mode – either both UEFI or both BIOS. So make sure when you are doing a clean install of the second OS it must boot via same mode as the already installed OS.